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EXECUTIVE SUMMARY 


Many youth in America are not on track for labor market success. One factor that increases the 
risk of poor labor market outcomes among these youth is dropping out of school (Rumberger 
2020). Youth who drop out of school are at greater risk for job instability and for lower long- 
term earnings (Hair et al. 2009). They are also more likely to struggle with mental health and 
substance abuse issues (Maynard et al. 2015). These challenges are compounded for youth who 
have early involvement with the juvenile or criminal justice systems. Even low levels of 
involvement can disrupt school attendance and increase the likelihood of dropping out of school 
(Kirk and Sampson 2013; Hjalmarsson 2008). Additional collateral consequences—including 
restrictions on financial aid, employer discrimination, and occupational licensing restrictions— 
also create barriers to future labor market success (Simpson and Holthe 2018). Youth with prior 
involvement in the justice system need targeted support to overcome these barriers (Office of 
Juvenile Justice Delinquency Prevention 2000). 


The National Guard Youth ChalleNGe (YC) Program is an evidence-based program for helping 
youth who have dropped out of high school get back on track. An evaluation of this program for 
youth ages 16 to 18 found that three years after program entry, YC participants were more likely 
than the control group to have obtained a GED or high school diploma, earned college credits, 
and be employed (Millenky et al. 2011). The YC model includes a 20-week, community-based 
residential program followed by a year of post-program mentoring that aims to build youth 
confidence and maturity, teach practical life skills, and help youth obtain a high school diploma 
or GED. Building on this successful model, the Employment and Training Administration (ETA) 
of the U.S. Department of Labor (DOL) funded YC programs in three states, to expand their YC 
programs to include more court-involved youth and to create a follow-on residential occupational 
training program called Job ChalleNGe (JC). 


DOL’s Chief Evaluation Office (CEO), in partnership with ETA, contracted with Mathematica 
and its subcontractors, Social Policy Research Associates and MDRC, to evaluate the JC grants. 
The evaluation examined the implementation of these grants and the outcomes for youth 
participants related to postsecondary education, employment, and criminal justice involvement in 
the two years following program involvement. This report describes our findings and presents 
lessons from the experiences of the three grantees and participating youth. 


Overview of the Job ChalleNGe grants and the evaluation 


In 2015, DOL issued a total of $12 million in JC grants to existing YC programs in Georgia, 
Michigan, and South Carolina. YC is authorized by the National Guard and is funded through a 
cooperative agreement between the U.S. Department of Defense and each state’s National 
Guard. The JC grants included 3 months of planning and 36 months of service delivery during 
which grantees were expected to serve six cohorts of youth in the JC program. DOL had two 
explicit goals for the JC grants: (1) to expand the target population of YC to include more youth 


xi 
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who have been involved with the courts and (2) to add a residential occupational training 
component, known as JC, that would be available as an option for YC graduates (DOL 2015). 


e Serving court-involved youth. The goal to expand services to more court-involved youth 
aimed to make a difference in the lives of youth who face additional barriers to education and 
employment because of their court involvement. Although DOL did not set an explicit target 
for the increase in the number of court-involved youth to be served by YC, each grantee had 
a target enrollment of 300 youth for the JC program with a goal of 50 percent of the JC youth 
being court-involved. 


e Providing occupational training. The core component of JC was a “robust vocational 
experience” that took place during the 20-week residential component of JC. Grantees 
partnered with community or technical colleges to deliver occupational skills training, 
individualized career and academic counseling, work-based learning opportunities, and 
leadership development opportunities. 


Exhibit ES.1 presents a typical path through YC and JC. Before the start of YC, youth accepted 
into the program participated in a two-week Pre-ChalleNGe phase. Youth who completed the 
Pre-ChalleNGe phase were formally enrolled into the program as cadets and transitioned to the 
20-week residential phase. The YC curriculum is structured to promote positive youth 
development with youth spending the largest share of each day in the education courses to gain a 
secondary education credential. The daily schedule is highly structured, and youth are closely 
supervised by staff at all times. After the YC residential phase, youth who were interested and 
eligible for JC could then complete an additional 20-week residential phase at the JC program 
location. Both programs were followed by a nonresidential phase that included structured 
mentoring and continued contact with ChalleNGe staff. In some cases, youth completed YC, 
returned home, and then enrolled in JC at a later date. 


Although each of the three grantees operated its YC and JC programs on a slightly different 
schedule, they typically enrolled youth twice per year, with each group of youth forming a 
cohort. Thus, the grantees were expected to serve six cohorts of youth in the JC program, each 
with about 50 youth, for a total of 300 youth during the entire grant period. 
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Exhibit ES.1. YC and JC typical program timeline 


Pre-ChalleNGe Youth ChalleNGe Nonresidential 
2 weeks residential phase phase 
20 weeks of Structured 
Youth on-site mentoring 
programming for up to one 
ChalleNGe year 
Job Job ChalleNGe Nonresidential 
residential phase phase 
ChalleNGe 20 weeks of 6-month 
on-site follow-up 
programming period 


Research questions. The JC evaluation is an implementation and outcomes study designed to 
provide a comprehensive picture of how the grantees implemented the DOL JC grants and to 
examine the outcomes of youth participants.! The evaluation addresses six main research 


questions: 

1. How was the Youth ChalleNGe program implemented under the Job ChalleNGe grant? 

2. How did the programs recruit and select youth for Job ChalleNGe? 

3. How was the Job ChalleNGe program implemented? 

4. How did youth in Youth ChalleNGe and Job ChalleNGe experience the post-residential 
phase? 

5. What were the outcomes of Youth ChalleNGe and Job ChalleNGe participants? 

6. What can we learn from these grants about possible program models to serve at-risk youth? 


Data sources. Our analysis draws on a mix of quantitative and qualitative data that bring 
together information at different points in time to address the evaluation’s research questions. 


Site visit data. Site visits to all three grantees were conducted twice during program 
implementation. Site visits included interviews with YC and JC administrators, staff, 
partners, and employers, observations of program activities, as well as focus groups with YC 
and JC youth. 


1 


In 2015, CEO instructed Mathematica and its subcontractors to conduct an assessment of whether a rigorous impact 


study would be feasible. Based on this assessment, DOL determined that an impact study was not advisable, because 
programs could not generate sufficient oversubscription of youth to support the random assignment of some youth 
into a control group. Since the JC program recruited from a fixed pool of YC graduates, it was not possible to 


Ss 


ubstantially increase recruitment. 
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e Survey data. A background information form administered to YC participants included 
youth demographics and history of delinquent behavior. Follow-up surveys of JC participants 
were conducted 16 to 23 months after the youth started the JC program. Survey responses 
included information on youth experiences in JC, service receipt, and employment outcomes. 


e Administrative data. National Student Clearinghouse data provided education outcomes 
including their postsecondary enrollment and credentials. Administrative data from state 
criminal justice agencies provided measures including arrests, convictions, and offense type. 


e Other sources of data. Individual-level program records from the YC and JC programs were 
used to assess representativeness of the youth who consented to participate in the evaluation. 
Grantee performance reports were used to describe aggregate enrollment in YC and JC, 
participant characteristics, program completion, and program enrollment and completion for 
court-involved youth across all cohorts within a program. 


Methodology 


To answer the study’s research questions, we conducted a descriptive analysis to explore how JC 
was implemented and to investigate JC participants’ education, employment, and criminal justice 
outcomes in the two years following involvement with the program. Our mixed-methods 
approach blended the qualitative data on program implementation with quantitative data on 
youth’s experiences and outcomes. 


Our quantitative analysis focused on a pooled analysis of the three JC grantees. The primary 
analyses present tabulations from the quantitative data sources to provide insights into the 
characteristics of participants, services and programs they received, and their outcomes. We also 
performed subgroup analyses on court-involved youth relative to non-court-involved youth. 
Because our analyses rely on quantitative data collected only from youth consenting to study 
participation and who filled out relevant surveys, we weighted all estimates to match the 
observable characteristics of the population of YC and JC participants. 


Findings 


DOL had two distinct goals for the JC grants: to improve youth education and employment 
outcomes by (1) providing more court-involved youth with access to an evidence-based youth 
program and (2) implementing a new occupationally-intensive program for YC graduates to 
further support youth development and prepare youth for the labor market. For grantees, the 
primary focus was the significant undertaking of establishing the new JC program. 


Youth ChalleNGe 


e JC grant recipients did not substantially change their YC recruitment or service 
delivery strategies. Grantees continued to use previous recruitment strategies that attracted 
some court-involved youth. Since grantees were not tracking court involvement prior to the 
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grant, we are not able to determine whether enrollments of court-involved youth changed 
under the grant. 


Some staff raised concerns about the potential stigma associated with the term “court - 
involved.” Reacting to these concerns, programs did not focus on court-involved youth in 
public marketing and did not distinguish between court-involved and non-court-involved 
youth within YC to avoid stigmatizing youth among their peers. 


Grantees found it challenging to reconcile DOL’s focus on court-involved youth with 
the existing Department of Defense guidance on YC eligibility criteria. The YC programs 
continued to follow Department of Defense guidance requiring that applicants have never 
been convicted of a felony and have no legal action pending. 


Court-involved youth had less education at enrollment and higher rates of prior 
delinquent behavior compared to other YC participants. Slightly higher portions of 
court-involved youth were male and had a child. They were more likely to report having been 
suspended from school or used drugs recently. 


Job ChalleNGe 


Recruitment and enrollment 


Grantees used a range of strategies to recruit YC youth into JC. Methods included 
starting recruitment early, such as at YC enrollment, maintaining outreach efforts to potential 
JC recruits over the course of YC, and engaging families to encourage youth to participate in 
JC. The size of the YC feeder program determined the ease of participant recruitment. The 
grantees with smaller YC feeder programs could not be selective about JC participants. 


JC grantees programs recruited and enrolled 905 youth from January 2016 to 
December 2018. The grantees met the overall enrollment target of 900 participants with JC 
enrollment of 333 in Georgia, 301 in Michigan, and 271 in South Carolina. 


Grantees did not meet the DOL performance target of 50 percent of court-involved 
youth. Based on the performance data grantees reported to DOL, 44 percent of JC 
participants were court-involved youth. Programs reported that they prioritized court- 
involved youth in the JC application process, but only one of the JC programs consistently 
had more applications than available program slots. 
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Exhibit ES.2. JC participant snapshot 


Participant Snapshots: 
Who Enrolled in Job ChalleNGe? 


Characteristics of Job ChalleNGe Participants Before Entering 
Youth ChalleNGe Program 


16 years old or younger Non-Hispanic, Black 


Age Race/ethnicity 


SR 


span 


i 


Court-involvement and Prior employment PS 
school disciplinary issues : ; 

a 7 ==: 30% » employed directly before 
36% » have any court involvement starting YC 
74% *® suspended from school 46% » ever had a paying job for at least 

3 months 
Other statistics 
G 5 > = c-) bs 

=— @® ® y § 

* 68% free «<1% “ <1% * 21% “5% “1% 
or reduced- married have a ever received unstable foster care 
price lunch child special housing involvement 

education 
services 


Source: Background information form collected from 304 youth in the three participating Job ChalleNGe C) programs located in Georgia, 
Michigan, and South Carolina. Analysis includes youth who enrolled in JC between July 2017 and July 2018 and who filled out a background 
information form. Youth were categorized as having court involvement if at the time of entering YC they reported ever being arrested, 
found guilty of a status offense, convicted of a crime, spent time in a juvenile or adult detention facility, or were on probation or parole. 
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Program implementation and service delivery 


All three grantees successfully established new occupationally-focused, residential 
programs for youth. The three JC programs used different housing environments, ranging 
from a military base to a rural retreat center. When selecting the location, administrators 
considered distances to local college partners and the YC program, as well as the availability, 
quality, and cost of the residential space. Administrators and staff reported that the residential 
location had implications for coordination with YC, staffing flexibility, transportation costs, 
and youth experience. 


Job ChalleNGe provided access to intensive occupational training and supplementary 
education. JC programs enrolled youth in community colleges where they took general 
education courses and participated in certificate-based vocational training programs. Youth 
who lacked a high school diploma or GED also participated in secondary education services. 
More than 95 percent of youth received some education service. 


Partnerships with community colleges were key to providing occupational training 
programs. JC programs formed partnerships with community colleges to provide 
occupational training and other related program services. All participants received some 
occupational training, and 73 percent received an occupational training certification. 


Over the course of implementation JC programs tightened their discipline models and 
provided more supervision of youth than initially planned. Although all three JC 
programs initially gave youth substantially more freedom than they had in YC, staff reported 
that many youth did not have the self-discipline to meet their program goals. Providing more 
intensive supervision required more staffing than originally planned. 


Outcomes of Job ChalleNGe participants 


Two-thirds of JC participants enrolled in postsecondary education within six months of 
completing YC, but few youth persisted (Exhibit ES.3). Despite the high postsecondary 
enrollment rate during the typical JC program period, only 10 percent of participants were 
enrolled in post-secondary education at the one-year mark following YC completion, 


Across the three sites, 86 percent of JC participants were involved in a productive 
activity approximately 14 months after JC (Exhibit ES.3). This finding is driven by the 
fact that 81 percent of JC participants were employed at the time of the survey. Average 
weekly earnings for youth were $379, and nearly two-thirds reported that JC helped to 
prepare them for their job. 


Post-program involvement with the justice system was relatively limited (Exhibit ES.3). 
Within one year of YC completion, eight percent of JC participants were arrested and 5 
percent were convicted of a new charge. 
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Exhibit ES.3 Key outcomes for JC participants (reported in percentages unless otherwise 
specified) 


Court- Not court- 

Outcomes ance) hYZ=Xe| ance) hYZ=Xe| p-value® 
Any productive activity?> 86 81 88 
Postsecondary education 

Enrolled within six months of YC° 67 66 68 

Enrolled one year following YC° 10 5 13 Zs 
Employment 

Currently working? 81 78 82 

Average weekly earnings (dollars)? 379 394 374 
Military 

Currently enlisted? 13 10 16 
Criminal justice involvement 

Arrested within one year of YC¢ 8 14 5 es 

Convicted within one year of YC4 5 9 2 ee 
Sample size (NSC and CJ) 304 103 194 
Sample size (FUS) 150 53 95 


Source: Weighted data from the background information form, follow-up survey, and administrative records. 

Notes: Analysis sample includes YC participants in Cohorts 4-6 who completed the background information form. 
Youth were categorized as having court involvement if at baseline they reported ever being arrested, found 
guilty of a status offense, convicted of a crime, or spent time in a juvenile or adult detention facility, or if they 
were on probation or parole at the time of entering YC. 

aSource: Follow-up survey (FUS) weighted data. 

>Any productive activity is defined as employment, education, or military enlistment. 

°Source: National Student Clearinghouse (NSC) weighted data. 

4Source: Criminal justice (CJ) administrative weighted data. Arrests and convictions are for new offenses. 

°Statistical significance is estimated using t-tests to compare differences between court-involved youth and the not 

court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 


Job ChalleNGe service and outcomes for court-involved youth 


e Court- and non-court-involved youth received largely the same services. JC staff 
members did not differentiate the services they provided to youth based on whether the 
youth were court-involved. The only exception was that staff steered some youth with 
criminal backgrounds away from occupational training in fields in which it might be 
difficult for people with criminal records to obtain employment. 


e Court-involved JC participants had similar rates of involvement in a productive 
activity as non-court-involved participants (Exhibit ES.3). Court-involved participants 
had similar rates of post-program employment to non-court-involved participants. 
Although court-involved participants were equally likely to be enrolled in postsecondary 
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education during the JC period, by one year following YC completion, they were less 
than half as likely to be enrolled than non-court-involved participants. 


e Court-involved JC participants had higher rates of post-program justice system 
involved (Exhibit ES.3). Fourteen percent of court-involved participants were arrested in 
the year following YC, relative to only 5 percent of non-court-involved participants. 


Considerations for the future 


As a voluntary and free program that offered a combined 42 weeks of residential programming, 
Youth and Job ChalleNGe is a unique opportunity for young people to not only “get back on 
track” but also build their skills for a successful career. All three DOL-funded Job ChalleNGe 
grantees launched new residential programs and developed partnerships with community 
colleges to offer participants occupational training and access to a college experience. 


At the time of this report’s release, the Job ChalleNGe program continues in the three pilot sites, 
without funding from DOL, plus a few additional locations around the country. This study offers 
lessons that can help inform current and future programming for Job ChalleNGe. Although the 
outcomes for JC participants are encouraging, this study does not provide evidence on the 
program’s effectiveness. Understanding the impact of JC on the employment, education, and 
criminal justice outcomes of youth participants requires an impact study. However, aspects of the 
JC program, including small cohorts and enrolling from a fixed population of YC graduates, 
make a random assignment study difficult to conduct. As the number of JC programs increases, 
there may be additional opportunities to measure the program’s impact including evaluation 
designs that compare the outcomes of JC participants to the outcomes of similar youth enrolled 
in other YC programs that do not have access to JC. 
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I. INTRODUCTION 


Many youth in America are not on track for labor market success. One factor that increases the 
risk of poor labor market outcomes among these youth is dropping out of school (Rumberger 
2020). Youth who drop out of school are at greater risk for job instability and for lower long- 
term earnings (Hair et al. 2009). They are also more likely to struggle with mental health and 
substance abuse issues (Maynard et al. 2015). These challenges are compounded for youth who 
have early involvement with the juvenile or criminal justice systems. Even low levels of 
involvement can disrupt school attendance and increase the likelihood of dropping out of school 
(Kirk and Sampson 2013; Hjalmarsson 2008). Additional collateral consequences—including 
restrictions on financial aid, employer discrimination, and occupational licensing restrictions— 
also create barriers to future labor market success (Simpson and Holthe 2018). Youth with prior 
involvement in the justice system need targeted support to overcome these barriers (Office of 
Juvenile Justice Delinquency Prevention 2000). 


The National Guard Youth ChalleNGe (YC) Program has been shown to improve education and 
labor market outcomes for high school dropouts between the ages of 16 and 18 (Millenky et al. 
2011). The YC model includes a 20-week, community-based residential program followed by a 
year of post-program mentoring that aims to build youth confidence and maturity, teach practical 
life skills, and help youth obtain a high school diploma or GED. Building on this successful 
model, the Employment and Training Administration (ETA) of the U.S. Department of Labor 
(DOL) funded YC programs in three communities, to expand their programs to include more 
court-involved youth and to create a follow-on residential occupational training program called 
Job ChalleNGe (JC). 


DOL’s Chief Evaluation Office (CEO) contracted with Mathematica and its subcontractors 
Social Policy Research Associates and MDRC to evaluate the JC grants. The evaluation 
examined the implementation of these grants and the outcomes for youth participants. This report 
describes our findings and presents lessons from the experiences of the three grantees and 
participating youth. 


A. The Youth ChalleNGe program model and Job ChalleNGe expansion 
DOL’s JC grant builds on the established YC program model and lessons learned from the 
evaluation of YC. 

1. Youth ChalleNGe program 


YC is authorized by the National Guard and is funded through a cooperative agreement between 
the U.S. Department of Defense and each state’s National Guard. Since it was first piloted in 
1993, YC has grown to 41 programs across 30 states. YC programs typically enroll two cohorts, 
or “classes” of youth annually. Each YC class has approximately 100 youth. 
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The goal of YC is to build youth confidence and maturity, teach practical life skills, and help 
youth obtain a high school diploma or GED. YC’s numerous activities address its eight core 
pillars: (1) leadership and followership, (2) responsible citizenship, (3) service to community, (4) 
life coping skills, (5) physical fitness, (6) health and hygiene, (7) job skills, and (8) academic 
excellence. 


YC operates as a quasi-military environment, in which participants, known as cadets, live in 
barracks-style housing and a disciplined environment for about 20 weeks—the residential phase. 
Cadets wear their hair short and dress in military uniforms. The daily schedule is highly 
structured with almost no downtime, and cadets are closely supervised by staff at all times. 


Upon completing the residential phase, participants receive a year of structured mentoring 
designed to help them transition back into their communities. The mentoring program is 
distinctive, in that young people nominate their own mentors during the application process. YC 
initiates the mentoring relationship partway through the residential phase, after the staff screen 
and train the mentors. Mentors can be family members, neighbors, or other adults the youth and 
staff expect will provide a positive and consistent influence on the youth. During the post- 
residential phase, staff maintain contact with both YC graduates and their mentors at least 
monthly, working with mentors to help solve problems and monitor each participant’s progress. 


2. Evidence on the effectiveness of Youth ChalleNGe 


YC has been documented to improve outcomes for youth. The National Guard Youth ChalleNGe 
Evaluation (ChalleNGe Evaluation), a 10-site, random assignment evaluation of YC led by 
MDRC, found large positive impacts for youth in the program group (Millenky et al. 2011). 
About 3,000 young people entered the study in 2005 and 2006. Some were randomly assigned to 
the program group, which could enroll in YC, while others were randomly assigned to a control 
group that could not enroll in YC. Data from the YC management information system show that 
about 83 percent of the program group started the program, 68 percent completed the two-week 
assessment and orientation period and formally enrolled, and 53 percent graduated. The key 
findings are summarized in the box “Key findings from the ChalleNGe Evaluation.” 


Key Findings from the ChalleNGe Evaluation (Millenky et al. 2011) 
Three years after study enrollment, survey data showed that the program group had attained higher 
levels of education and achieved better labor market outcomes. Relative to the control group, the 
program group was: 
e More likely to have obtained a GED and to have earned college credits 
e More likely to be employed at the time of the survey 
e More likely to have higher earnings (the program group earned about 20 percent more) 
These impacts were not statistically different for youth with and without justice system involvement. 


Despite these early and positive impacts, follow-up interviews confirmed that even program graduates 
had difficulty gaining a foothold in college or the labor market. 
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The report concluded with a set of recommendations to enhance the post-residential phase of the 
program to help young people negotiate the difficult transition from the highly structured and 
supportive residential program to their communities. One recommendation was to build and 
provide a stronger vocational training component, either during or after the residential phase of 
the YC program. 


3. Job ChalleNGe grants 


In 2015, DOL issued a total of $12 million in JC grants to YC programs in Georgia, Michigan, 
and South Carolina (Exhibit I.1). The grants included 3 months of planning and 36 months of 
service delivery during which grantees were expected to serve six cohorts of youth in the JC 
program. 


Exhibit I.1. JC grantees 


Youth 
ChalleNGe No) om Of ar-1I(-1) (C(- Mmm Co) om Od ar-lI(-1\LeI=) Occupational 
Grantee name program name | program name location training partner 
National Guard Fort Stewart Georgia Job Fort Stewart, Savannah 
Youth Georgia Youth ChalleNGe Georgia Technical College 
ChalleNGe/Job ChalleNGe Program 
ChalleNGe Academy 
Academy 
wowamven, Michigan Michigan Youth Michigan Job Fort Custer Kellogg 
Department of ChalleNGe ChalleNGe Training Center, | Community 
Military and Academy Program Battle Creek, College (Regional 
“ikon Veterans Michigan Manufacturing 
Affairs and Training 
| Center) 
vy, South Carolina South Carolina South Carolina Aiken, South Aiken Technical 
i So Military Youth Youth ChalleNGe Carolina College 
<A Department ChalleNGe Academy POST 
_ Academy ChalleNGe 
Program 


DOL had two explicit goals for the JC grants: to improve youth education and employment 
outcomes by (1) expanding the target population of YC to include more youth who have been 
involved with the courts and (2) adding a residential occupational training component, known as 
JC, that would be available as an option for YC graduates (DOL 2015). 


e Serving court-involved youth. The goal to expand services to more court-involved youth 
aimed to make a difference in the lives of youth who face additional barriers to education and 
employment because of their court involvement. Substantial research suggests that to reduce 
the likelihood of recidivism and increase their chances for success, youth involved in the 
justice system need specific supports and interventions, such as enrollment in schooling or 
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job training programs, as well as access to housing and adult mentors (Beale-Spencer and 
Jones-Walker 2004). These supports, which are part of YC and JC, can make a difference in 
youth’s ability to find and keep jobs (Mears and Travis 2004). Although DOL did not set an 
explicit target for the increase in the number of court-involved youth to be served by YC, 
each grantee had a target enrollment of 300 youth for the JC program with a goal of 50 
percent of the JC youth being court-involved. 


e Providing occupational training. The core component of JC was a “robust vocational 
experience” that took place during the 20-week residential component of JC. Grantees 
partnered with community or technical colleges to deliver occupational skills training, 
individualized career and academic counseling, work-based learning opportunities, and 
leadership development opportunities. 


In addition to the explicit goals set by DOL, grantee staff viewed JC as a way to help prepare 
youth for life beyond YC. For example, staff discussed that JC aimed to allow youth more 
freedom than YC, so that youth were ready to make healthy choices on their own after leaving 
the residential program. Staff also viewed JC as an opportunity to arm youth with the skills they 
needed to succeed in school or work. 


For additional information on the specific grantees, see the program profiles in Appendix A. 


Exhibit I.2 presents a typical path through YC and JC. Before the start of YC, youth accepted 
into the program participated in a two-week Pre-ChalleNGe phase. Pre-ChalleNGe is a 
physically and psychologically demanding assessment and orientation period during which youth 
were introduced to the program’s rules and expectations; learned military bearing, discipline, and 
teamwork; and began physical fitness training. Youth who completed the Pre-ChalleNGe phase 
were formally enrolled into the program as cadets and transitioned to the 20-week residential 
phase. After the YC residential phase, youth who were interested and eligible for JC could then 
complete an additional 20-week residential phase at the JC program location. Both programs 
were followed by a nonresidential phase that included structured mentoring and continued 
contact with ChalleNGe staff. In some cases, youth completed YC, returned home, and then 
enrolled in JC at a later date. 


Although each of the three grantees operated its YC and JC programs on a slightly different 
schedule, they typically enrolled youth twice per year, with each group of youth forming a 
cohort. Thus, the grantees were expected to serve six cohorts of youth in the JC program, each 
with about 50 youth, for a total of 300 youth during the entire grant period. 
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Exhibit I.2. YC and JC typical program timeline 


Pre-ChalleNGe Youth ChalleNGe Nonresidential 
2 weeks residential phase phase 
20 weeks of Structured 
Youth on-site mentoring 
programming for up to one 
ChalleNGe Near 
Job Job ChalleNGe Nonresidential 
residential phase phase 
ChalleNGe 20 weeks of 6-month 


follow-up 
period 


on-site 
programming 


B. The Job ChalleNGe Evaluation 


The JC evaluation is an implementation and outcomes study designed to provide a 
comprehensive picture of how the grantees implemented the DOL JC grants and to examine the 
outcomes of youth participants.” The evaluation addresses six main research questions: 


1. How was the Youth ChalleNGe program implemented under the Job ChalleNGe grant? 
How did the programs recruit and select youth for Job ChalleNGe? 


2 
3. How was the Job ChalleNGe program implemented? 
4 


How did youth in Youth ChalleNGe and Job ChalleNGe experience the post-residential 
phase? 
5. What were the outcomes of Youth ChalleNGe and Job ChalleNGe participants? 


6. What can we learn from these grants about possible program models to serve at-risk youth? 


1. Study enrollment 


Each DOL grant funded JC services to six cohorts of youth, with the first cohort beginning the 
YC program in July 2015 and the final cohort beginning the JC program in July 2018 (Exhibit 


aot 2015, CEO instructed Mathematica and its subcontractors to conduct an assessment of whether a rigorous 
impact study would be feasible. Based on this assessment, DOL determined that an impact study was not advisable, 
because programs could not generate sufficient oversubscription of youth to support the random assignment of some 
youth into a control group. Even if recruitment efforts increased, the study would not have had enough sample size 
to answer the key research questions with sufficient statistical precision. 


(continued) 
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1.3). The implementation study included all six cohorts of YC youth and all six cohorts of JC 
youth.? The outcomes study included youth from the final three cohorts of YC and JC.* 


Exhibit I.3. Timing of study enrollment 


Youth ChalleNGe cohort 


A 
YC1 YCc2 YC3 YC4 Ycs YC6 
JC1 JC2 JC3 JC4 JC5 JC6 


Job ChalleNGe cohort 


Source: Staff interviews. 


Note: The timing of the cohorts was uniform across grantees. YC Cohorts 4-6 and JC Cohorts 4-6 are 
included in the outcomes study. 


Enrollment in the outcomes study began when youth enrolled in the program as they entered the 
residential phase of the program. The enrollment process included obtaining (1) youth consent 
for youth age 18 and (2) parent or guardian consent and youth assent for youth under the age of 
18. If the study team was not able to obtain their consent at entry into YC or their parent was not 
present to give consent at that time, a second attempt was made to enroll JC participants in the 
study at the start of the JC residential phase. Across all three grantees, 984 of the 1,460 YC youth 
participated in the outcomes study (meaning 67 percent of the YC youth consented to be in the 
study). Of the YC youth, 392 youth completed YC cohorts four through six and enrolled in JC, 
of which 304 JC youth participated in the outcomes study (meaning 78 percent of JC youth 
consented to be part of the study). To adjust for bias due to nonconsent, all analyses are weighted 
such that the observable characteristics of the sample are representative of the observable 
characteristics of the overall sample. 


2. Data collection 


Our analysis draws on a mix of quantitative and qualitative data from seven primary sources that 
bring together information at different points in time to address the evaluation’s research 
questions. Exhibit I.4 lists the programs and cohorts for which we obtained each individual-level 
data source. For a detailed description of each quantitative data source, see Appendix B. 


e Site visits to all three grantees were conducted twice during program implementation (fall 
2017 and spring 2018). Site visits included interviews with YC and JC administrators, staff, 


a7C study-cohorts YC1 through YC6 correspond to what the YC Academy refers to as Classes 45 through 50. 


4 Youth in the first three cohorts of YC and the first three cohorts of JC were not included in the outcomes study, 
because grant activities were underway before the study received approval from the Office of Management and 
Budget to start enrollment into the outcomes study. 
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partners, and employers, observations of program activities, as well as focus groups with YC 
and JC youth. In total, we conducted interviews with 59 administrators and staff across the 
three programs, staff from 11 partner organizations, and one employer partner from each site. 
We also conducted 3 focus groups with YC youth and 6 focus groups with JC youth. Across 
all 9 focus groups, 39 youth participated. Because youth were not randomly selected for the 
focus groups, and hence they do not present a representative perspective of all YC and JC 
participants, data collected from the focus groups are used to provide anecdotal support of 
findings based on other data sources; no findings in the report are based solely on the 
information from the focus groups. 


e Background information form. After obtaining consent from youth (or assent from youth 
under 18 with consent from their parent or guardian), the study team collected data on youth 
demographics, whether they were involved in the foster care system, history of delinquent 
behavior, and contact information through the background information form. This data was 
collected over the course of YC or at JC program entry, but the questions referred to 
experiences before YC. Overall, 984 youth in YC cohorts 4-6 filled out a background 
information form. 


e Program data. We collected individual-level program records from the YC and JC 
programs. Data included identified records for youth enrolled in the study and deidentified 
records for the remainder of program participants. We requested data on participant 
characteristics, services, and program completion. We used this data to assess 
representativeness of the youth who consented to participate in the evaluation. Due to 
variation in data availability and quality across programs, we were unable to conduct 
additional analyses to assess service receipt, duration of involvement, or program completion 
rates. 


e Follow-up surveys of JC participants were conducted 16 to 23 months after the youth 
started the JC program. One hundred fifty JC youth in JC Cohorts 4-6 completed the follow- 
up survey through the web (a 45 percent response rate for consenting youth in these cohorts). 
To address differences, all follow-up survey analyses are weighted so that the observable 
characteristics of the sample are representative of the observable characteristics of the full set 
of JC youth. For more details on the characteristics of respondents relative to nonrespondents 
and our calculation of survey weights, see Appendix C. Survey responses included 
information on youth experiences in JC, service receipt, and employment outcomes. We also 
conducted a short monthly text survey in the period between JC completion and the follow- 
up survey to explore whether this approach could provide periodic snapshots of employment 
and education status. The text survey sample included 113 youth from Cohorts 5 and 6 who 
indicated on the background information form that they were willing to receive text messages 
from the evaluation. Of these youth, only 12 responded to all six months of the survey, and 
only 32 responded to at least one month. See Appendix B for more details on survey data 
collection. Given the low response rate of the text-survey data collection, these data were 
used to create illustrative descriptions of youth employment, but no study findings are based 
on these data. 
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e National Student Clearinghouse (NSC). We collected NSC data to obtain education 
outcomes including their postsecondary enrollment and credentials. The NSC maintains a 
comprehensive administrative database on more than 99 percent of U.S. colleges and 
universities with information on student enrollment in post-secondary education, school 
characteristics, program type, degrees obtained, and program of study. The NSC does not 
contain information from institutions other than colleges and universities, such as trade 
schools or apprenticeship programs. For more details on the NSC, see Appendix B. 


e State criminal justice records. We collected administrative data on criminal justice 
involvement from the state agencies for youth who consented. Measures include arrests, 
convictions, and offense type. The administrative data are limited to adult offenses, although 
the age at which youth can be charged as adults varies across the three grantee states. 
Additionally, data received from each state varied in format and content; for example, some 
states did not provide information on arrests that did not result in a conviction, while other 
states did. For a description of the data provided by each state, see Appendix B. 


e Grantee performance reports from the quarter ending March 2019 were used to describe 
aggregate enrollment in YC and JC, participant characteristics, program completion, and 
program enrollment and completion for court-involved youth across all cohorts within a 
program. 


Exhibit 1.4. Quantitative data sources and cohorts included 


Youth ChalleNGe Mo) om OF at-11 (21) LOL) 


Data source 


Background information form YC 4-6 JC 4-6 
Program data YC 4-6 JC 4-6 
Follow-up survey None JC 4-6 
Text message survey 2 None JC 5-6 
NSC YC 4-6 JC 4-6 
State criminal justice data YC 4-6 JC 4-6 
Grantee performance reports YC 1-6 JC 1-6 


@The text message survey sample is limited to youth who gave permission to be contacted by text 
message. 


NSC = National Student Clearinghouse. 


3. Analysis 


To answer the study’s research questions, we conducted a descriptive analysis to explore how JC 
was implemented and to investigate JC participants’ education, employment, and criminal justice 
outcomes in the two years following involvement with the program. Our mixed-methods 
approach blended the qualitative data on program implementation with quantitative data on 


Job ChalleNGe Final Report Mathematica 


youth’s experiences and outcomes. Using both types of data provided a broader perspective on 
the program than reliance on a single type of information. 


We identified key themes and topics in the qualitative data collected during site visits using a 
two-stage coding analysis. In the first phase, we identified broad themes and takeaways on key 
topics related to the research questions. In the second stage, we took an inductive approach, 
whereby we developed more descriptive codes that could better capture the range of strategies 
that grantees used on a given topic. For example, if in the first phase we coded broadly for 
challenges encountered, in the second phase, we could have coded for specific types of 
challenges to assess whether certain challenges occurred across all three grantees or were 
reported at one or two grantees only. 


Our quantitative analysis focused on a pooled analysis of the three JC grantees. Pooling analyses 
across grantees increased the sample size and enabled more detailed analyses (for example, 
analyses on subgroups of participants). It also allowed us to present results for the universe of JC 
grantees. However, this approach may mask meaningful variation across grantees. Therefore, we 
present additional results for key analyses by grantee in Appendix E. 


Our primary analyses present tabulations from the quantitative data sources to provide insights 
into the characteristics of participants, services and programs they received, and their outcomes. 
We also performed subgroup analyses on court-involved youth relative to non-court-involved 
youth. Given that the grantees used different criteria to define which youth were court-involved 
(see Chapter II for more information), we identified which youth were or were not court- 
involved using self-reports on justice system involvement prior to the YC program from the 
background information form. This process enabled us to measure court involvement 
consistently across grantees. To identify whether there were differences in experiences and 
outcomes between subgroups of youth, such as those who were and were not court-involved, we 
also conduct statistical tests based on chi-squared analyses of the tabulations. We present 
statistical significance using thresholds of 1, 5, and 10 percent. Statistical tests represent the 
statistical significance if we consider our true population to be a hypothetical universe of all 
potential JC participants. 


Because our analyses rely on quantitative data from youth who consented to the study and 
completed relevant surveys, our sample does not include the full population of YC and JC 
participants in the three grantee programs. This approach may lead to bias if the youth who 
participate in the study are not representative of the full population of program youth. To address 
potential bias due to nonconsent and nonresponse, we weighted all estimates to match the 
observable characteristics of the population of YC and JC participants. This process is described 
in more detail in Appendix C. 


Findings from this study reflect the implementation of JC under the DOL JC grants. These 
results represent the experiences of three programs and may not be generalizable to other YC 
programs that might implement a JC program. For example, if other YC programs implement a 
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JC program with other sources of funding, they may not have the same required program 
services or a focus on court-involved youth. This study also does not present a causal analysis of 
the impact of JC on participants. There are many factors that impacted JC youth, and we are 
unable to identify how each factor impacted program experiences and participant outcomes. 


The remaining chapters of the report map to the research questions as follows: 


e Chapter II describes the extent to which grantees adapted their YC programs under the JC 
grant, including whether and how they recruited more court-involved youth (Research 
Question 1). 


e Chapter II relays how JC programs recruited and selected youth and describes JC 
participants, including how JC participants compare with YC-only participants and JC 
program attrition (Research Question 2). 


e Chapter IV describes the JC residential environment and housing, JC staffing, and the 
disciplinary approaches staff used (Research Question 3). 


e Chapter V details the educational, occupational, and nonacademic services JC participants 
received as well as youths’ connections to the program during the post-residential phase 
(Research Questions 3 and 4). 


e Chapter VI presents post-program outcomes for JC participants overall and compared with 
outcomes for YC-only youth (Research Question 5). 


e Chapter VII summarizes the key findings, lessons learned from the implementation of JC, 
and considerations for other programs serving youth (Research Question 6). 


e The appendices included in this report include one-page profiles on each of the JC grantees 
(Appendix A), descriptions of the quantitative data sources (Appendix B), information on the 
weighting for study nonconsent and survey nonresponse (Appendix C), regression analyses 
of post-program outcomes (Appendix D), grantee-specific results (Appendix E), follow-up 
survey estimates of criminal justice outcomes (Appendix F), and estimates of key participant 
outcomes using the grantee-defined measures for whether a youth had court and/or justice 
involvement (Appendix G). 
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ll. IMPLEMENTATION OF YOUTH CHALLENGE UNDER THE 
JOB CHALLENGE GRANT 


A central goal of the JC grants was to increase enrollment of court-involved youth into YC. 
From January 2016 to December 2018, the three YC programs that received JC grants reported 
enrolling 3,095 youth into YC, 38 percent of which had some form of court or justice 
involvement. This chapter describes how YC program adjusted under the JC grants, and in 
response to the increased focus on court-involved youth, with regard to YC program recruitment 
and enrollment, YC services, the characteristics of YC participants, and YC program completion. 
The findings in this chapter are based primarily on information shared during interviews with YC 
program staff, focus groups with youth, and data collected on youth through the background 
information form. 


Key Findings 


e JC grant recipients did not substantially change their YC recruitment or service 
delivery strategies. Grantees continued to use previous recruitment strategies that attracted 
some court-involved youth. However, grantees worked to better identify and document court 
involvement among youth already enrolled in YC. 


e Grantees created their own definitions of “court-involved.” Some grantees used more 
expansive definitions that included youth in foster care or youth with family court cases, while 
others defined court involvement more narrowly as youth with criminal offense histories. 


e Some staff raised concerns about the potential stigma associated with the term “court- 
involved.” Reacting to these concerns, programs did not focus on court-involved youth in 
public marketing and did not distinguish between court-involved and non-court-involved youth 
within YC to avoid stigmatizing youth among their peers. 


e Grantees found it challenging to reconcile DOL’s focus on court-involved youth with 
the existing Department of Defense guidance on YC eligibility criteria. The YC programs 
continued to follow Department of Defense guidance requiring that applicants have never 
been convicted of a felony and have no legal action pending. 


e YC programs enrolled youth who were disconnected from school and at risk of 
unstable employment in the future. Most youth entered YC at ages 16 or 17, from low 
income households, and with only some high school education. Across the three YC 
programs, data collected at baseline showed an average of 41 percent of participants had 
some form of court involvement. 


e Court-involved youth had a similar racial/ethnic composition to other YC participants 
but had less education at enrollment and higher rates of prior delinquent behavior. 
Slightly higher portions of court-involved youth were male and had a child. They also had 
completed fewer years of education and were more likely to report having been suspended 
from school or used drugs recently. 
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A. Defining court involvement 


Increasing enrollment of court-involved youth was a central objective of the JC grants. To meet 
this objective, grantees needed to define what constituted court involvement, but grantees found 
it challenging to operationalize this definition. In the original grant funding announcement, DOL 
defined court-involved youth as youth who had come into contact with the juvenile justice 
system through a status offense° or by committing a delinquent act and who had not been 
convicted as adults (DOL 2015). During site visit interviews, JC and YC staff members reported 
confusion around the types of court and justice system involvement that classify a youth as 
“court-involved” but do not lead to their exclusion from YC based on Department of Defense 
(DoD) eligibility guidelines. JC staff members generally used a broad definition for JC 
enrollment purposes and included any interaction with juvenile courts, regardless of 
adjudication, as indicating court involvement. This approach meant that YC enrollment remained 
largely unchanged under the JC grants. 


There was some variation in how the programs defined court involvement. The definitions used 
by Michigan and South Carolina were broader than Georgia’s definition; they counted 
noncriminal interactions with the courts—such as through family courts governing foster care, 
adoption, or divorce—as court involvement. All three YC programs also set individual 
exclusionary criteria. Georgia, for example, would not accept youth on probation without a letter 
indicating that probation would be suspended during enrollment in YC. In South Carolina, 
eligibility criteria excluded anyone with a violent offense, such as assault or rape. The Michigan 
YC program screened out youth with involvement in the justice system who appeared to be 
interested in YC only as a way to avoid court sanctions; youth had to demonstrate they were 
intrinsically motivated to be in the program. See Exhibit II.1 for each grantee’s definition of 
court-involved youth. For estimates of court-involvement rates and select outcomes by court- 
involvement based on grantee definitions in program data, see Appendix G. 


Exhibit Il.1. Grantee-specific definitions of court-involved youth 


LeT-Yo) ge] =) Michigan South Carolina 

Any juvenile in the court system, Anyone who was formally Anyone who was involved with 

excluding youth in the foster adjudicated, arrested, or the juvenile system, adult 

care system. Applicants who charged. Also included anyone — system, or social services 

were on probation were required in the foster care system or (homeless or foster youth). 

to obtain a letter stating their family court system. Youth with Youth with pending juvenile 

probation would be suspended open or pending cases and charges were eligible. Youth 

during participation. youth on probation were not with violent offenses were not 
eligible. eligible. 


Source: JC staff interviews. 


> A status defense is defined as a noncriminal act that is a violation of the law for minors. Examples include running 
away from home and underage use of alcohol. 
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B. Recruitment and enrollment 


Broadly, YC programs have established statewide recruitment and enrollment procedures. YC is 
marketed as a program that aims “to intervene in and reclaim the lives of at-risk youth to produce 
program graduates with the values and skills necessary to excel as adults.” Each state operated its 
own application process, with youth required to submit a written application and participate in an 
interview. The overall eligibility criteria are established by the National Guard (see box “YC 
eligibility criteria’). 


1. Recruiting court-involved youth for Youth ChalleNGe 


YC staff members reported in interviews with study team members that they had a history of 
serving youth with court involvement 
prior to the JC grant. As a result, staff 
did not perceive a need to make 
major changes to their YC 


YC eligibility criteria 


recruitment strategies as part of the e Committed to making a change in their life 
JC grant implementation. YC staff e Between 16 and 18 years of age 

also expressed that they did not add e U.S. Citizen or permanent resident 

court involvement to general e Educationally at risk or high school dropout 


marketing materials because they did 
not want YC to develop a reputation 
as a program for court-involved 
youth due to a perceived stigma by 
the larger public. 


e Voluntary enrollment 
e Drug-free (will be drug tested) 


e Physically and mentally capable to participate in 
the program with reasonable accommodations 
for physical and other disabilities 


e Not currently on parole or probation for other 


Staff from two YC programs reported 
maintaining previous relationships 
with justice system partners. South 


than juvenile status offenses, not awaiting 
sentencing, not under indictment, accused or 
convicted of a felony; cannot have any pending 


court dates once the program starts 
Source: DoD Instruction 1025.8 (2002) 


Carolina staff members reported that 
they built relationships with the 
Department of Juvenile Justice and 
family courts prior to the start of their 
JC grant, and continued to maintain these relationships during the grant. Similarly, Michigan 
continued to collaborate with local judges who had previously made referrals to YC as part of the 
court’s diversion efforts. Recruitment staff at these YC programs discussed how they were 
continually conducting outreach and trying to foster new connections with the justice system. 
However, these changes were already underway at the start of JC and were not driven by the JC 
grants. 


Staff members at all three YC programs made administrative changes to their application and 
intake procedures to gather additional information on court involvement. YC programs already 
asked applicants about their criminal histories and, if needed, conducted a criminal background 
check to comply with National Guard eligibility criteria, but they collected and tracked different 
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information to comply with the DOL requirements. YC staff reported changing the amount of 
criminal justice system information that they gathered on prospective participants, including 
where and how they recorded when youth self-identified as court-involved so they could share 
this information with JC program staff members. For example, in South Carolina, YC staff 
members began to document any justice-system involvement, including when a court had 
ordered a young person to enroll in the YC program. Michigan also became more intentional 
about asking youth at YC enrollment whether they had any court involvement and developed 
questions to probe for interactions with the justice system. For example, they asked applicants 
whether they had ever been handcuffed or had records expunged. 


We are not able to measure whether the enrollment of court-involved youth increased under the 
JC grant. Prior to the JC grant, YC did not track the number of court-involved youth who 
enrolled in the program. The ChalleNGe Evaluation conducted by confirms that YC was already 
serving youth with court involvement. In that study, 34 percent of youth reported having ever 
been arrested, and 19 percent reported having ever been convicted (Millenky et al. 2010). 


C. Youth ChalleNGe services under the Job ChalleNGe grant 


YC staff indicated during site visit interviews that they made very few, if any, changes to their 
YC program, including services and disciplinary policies, as a result of the JC grants. In Georgia 
and Michigan, only a few staff members knew which youth were court-involved. For instance, in 
Georgia, interview respondents reported that they would not vy. 

know which youth were court-involved outside of those who “I don’t know the difference in 
had to visit their probation officer or social worker or used 
their probation officer as a mentor. 


working with them [justice- 
involved kids] or a kid who’s 
never even smoked a cigarette. | 


: ae a couldn't tell you.” 
Overall, there was little distinction made within the 
—JC staff | 


programs between court-involved and non-court-involved 
youth. Staff members reported concerns that drawing 
attention to the court-involved youth population might negatively affect the youth themselves 
and the perception of them by other youth. For example, in Michigan, YC staff members 
described the fear that using the term court-involved or asking youth to self-identify as court- 


involved would stigmatize and negatively label the program 
W... say ‘atrisk youth’ like and youth, affecting their perceptions of themselves and 
we're criminals. Everybody at others’ perceptions of them. Staff at all sites also reported 
risk. They make it seem like it's that they did not perceive a difference between the court- 


a juvenile [justice program] or 


eta ‘ involved and non-court-involved youth, and therefore 
something like that. 


disliked the categorization. Youth in focus groups also 
reported disliking the term “at-risk” or being identified as 
court-involved, because not all of them had been involved 
with the justice system or felt the distinction was relevant to their lives. 


—YC participant 


14 


Job ChalleNGe Final Report Mathematica 


D. Snapshot of Youth ChalleNGe participants 


The three YC programs enrolled 3,095 YC participants during implementation of the JC grants. 
Among the 984 youth that completed a study baseline information form, 80 percent were male 
and more than half (53 percent) were 16 at program entry. Half of participants (50 percent) 
identified as non-Hispanic black, 32 percent as non-Hispanic white, and the remainder as 
Hispanic, mixed race, or another race. For a snapshot of the characteristics of YC participants, 
see Exhibit II.4. 


Many YC youth face risk-factors or barriers to continuing their education or finding and 
maintaining stable employment, such as low levels of education, limited work history, and for 
some youth, having prior justice involvement. Only 26 percent of youth had completed grade 11 
or higher, and almost none of the youth reported having a GED or high school diploma. Youth 
did have some labor market experience, with 44 percent reporting that they held a job for at least 
three months. Only 4 percent of youth reported unstable living situation, 3 percent had children, 
and less than 1 percent were married. Most youth came from low income backgrounds, with 
almost 75 percent reporting eligibility for free or reduced-price lunch in the two years prior to 
YC enrollment. Along most dimensions, the YC participant characteristics were similar across 
the three programs, but there was substantial variation in the racial composition of the three 
programs (see Appendix E). 


Forty-one percent of YC participants reported some court involvement prior to program entry 
(Exhibit II.2). This rate is estimated based on youth self-reports of ever being arrested, found 
guilty of a status 
offense, convicted of 
a crime, or detained in | 40% 
a juvenile or adult 

detention facility, or if | %% 
they were on 

probation or parole at | 7°” 


29 
19 18 = 
the time of entering 14 
YC.° Among the YC — ; rt 
rticipants, 2 
participants, 29 om aaa 


Exhibit Il.2. Types of YC youth court-involvement 


41 


percent reported that Any court Any status Ever Ever Ever Ever On probation 

involvement offense arrested or convicted _— detained in detained _—_ or parole at 

they were ever taken into ajuvenile inanadult YC entrance 
custody facility facility 


arrested or taken into 
custody, and 18 Source: Background information form weighted data. 


Notes: Analysis sample includes YC participants in Cohorts 4-6 who filled out 


percent reported 5 the background information form. N = 984. 


conviction. Seventeen 
percent had been detained in a juvenile facility, and 5 percent in an adult facility. The overall rate 


as apply a uniform definition of court involvement across the three YC programs, these statistics and those shown 
in the snapshot above define court-involvement based on the background information forms collected at the time 
of study enrollment. 
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of self-reported court involvement is slightly higher than the 36 percent reported on grantee 
performance reports submitted to DOL. The difference could reflect either differences across 
cohorts (the background information forms were collected for Cohorts 4-6 while the 
performance reports include all six cohorts) or differences in how program staff and youth define 
and report court involvement. 


When comparing court-involved and non-court-involved youth, demographic characteristics 
were relatively similar (Exhibit II.3), but court-involved youth had lower levels of education at 
enrollment (Exhibit II.5) and higher rates of self-reported delinquent behavior (Exhibit II.6). In 
particular, 88 percent of court-involved youth had a prior school suspension compared to 74 
percent of non-court-involved youth. There were also significant differences in prior drug use. 
Seventy percent of court-involved youth had used marijuana in the six months prior to YC 
enrollment compared to 43 percent of non-court-involved youth. Similarly, 35 percent of court- 
involved youth used another drug in the six months prior to YC enrollment compared to 15 
percent of non-court-involved youth. 


Exhibit Il.3. Baseline characteristics of YC youth overall and by status of court 
involvement (reported in percentages) 


Court- Not court- 
Characteristics Tanyce) hYC=re | ance) hYZ=Xe| p-value? 


Male 80 84 78 * 
Race and ethnicity 
Hispanic 8 8 9 
Non-Hispanic, black 50 52 49 
Non-Hispanic, white 32 32 31 
Non-Hispanic, other race 10 7 11 
Involved in foster care> 1 2 1 
Free and reduced-price lunch status® 73 73 72 
Unstable housing® 4 5 3 
Ever received special education services 19 20 18 
Married 0 0 0 
Has a child 3 6 2 ais 
Sample size 984 391 566 


Source: Background information form weighted data. 


Notes: Analysis sample includes YC participants in Cohorts 4-6 who filled out the background 
information form. Youth were categorized as having court involvement if at baseline they 
reported ever being arrested, found guilty of a status offense, convicted of a crime, or spent 
time in a juvenile or adult detention facility, or if they were on probation or parole at the time of 
entering YC. 


4Statistical significance is estimated using chi-squared difference tests to compare differences between 
the court-involved youth and the not court-involved youth. *p < 0.1; ** p < 0.05; *** p < 0.01. 

‘Foster care and housing status were self-reported at the time of the background information form 
collection. Free and reduced-price lunch status was self-reported based on the two years prior to 
background information form collection. 
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Exhibit ll.4. YC participant snapshot 


Participant Snapshots: 
Who Enrolled in Youth ChalleNGe? 


Characteristics of Youth ChalleNGe Participants Before Program Entry 


Age Race/ethnicity 


Court-involvement and 
school disciplinary issues 


Prior employment <= 


28% » employed directly before 


41% » have any court involvement starting YC 
80% » suspended from school 44% » ever had a paying job for at least 
3 months 


Other statistics at time of enrollment 


Zoos FS f 


4 73% 40% 43% 419% 4“ 4% 41% 
free or married havea ever received unstable foster care 
reduced-price child special housing involvement 
lunch education 
services 


Source: Background information form collected from 984 youth in the three participating Youth ChalleNGe (YC) prograrns located in Georgia, 
Michigan, and South Carolina. Analysis includes youth who enrolled in YC between January 2017 and January 2018 and who filled out a back- 
ground information form. Youth were categorized as having court involvernent if at the time of entering YC they reported ever being arrested, 
found guilty of a status offense, convicted of a crime, spent time in a juvenile or adult detention facility, or were on probation or parole. 
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Exhibit II.5. Baseline education and employment of YC youth overall and by status of 
court involvement (reported in percentages) 


Court- I Ko) mee) 0) ge 


Characteristics involved Tan Zed hc=xe| p-value? 


Educational attainment 


Last grade completed in school ee 
8th grade or below 8 7 8 
9th grade 26 29 23 
10th grade 40 43 38 
11th grade 22 17 26 
12th grade 4 4 


High school diploma or GED 


Employment 


Employed directly before YC 28 30 27 
Ever had a paying job for 2 3 months 44 46 42 
Sample size 984 391 566 


Source: Background information form weighted data. 


Notes: Analysis sample includes YC participants in Cohorts 4—6 who filled out the background 
information form. Youth were categorized as having court involvement if at baseline they 
reported ever being arrested, found guilty of a status offense, convicted of a crime, or spent time 
in a juvenile or adult detention facility, or if they were on probation or parole at the time of 
entering YC. Analyses on the full sample include court-involved youth, non-court-involved youth, 
and youth with missing information on court-involvement. Twenty-seven sample members were 
missing information on court involvement. 


4Statistical significance is estimated using chi-squared difference tests to compare differences between 
the court-involved youth and the not court-involved youth. *p < 0.1; ** p < 0.05; *** p < 0.01. 
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Exhibit Il.6. Self-reported delinquent behavior and justice-system involvement of YC 
youth overall and by status of court-involvement (reported in percentages) 


Court- Not court- 
Characteristics Tanyce) YC=xe | involved p-value? 
Ever suspended 80 88 74 isis 
Used marijuana in past six months 54 70 43 i 
Used another drug in past six months 23 35 15 —_ 
Any court involvement A 100 
Ever arrested or taken into custody 29 72 
Any status offense> 19 47 
Ever convicted 18 46 
Ever detained in a juvenile facility 17 A 
Ever detained in an adult facility 5 13 
On probation or parole at YC entrance 14 34 
Sample size 984 391 566 


Source: Background information form weighted data. 


Notes: Analysis sample includes YC participants in Cohorts 4-6 who filled out the background 
information form. Youth were categorized as having court involvement if at baseline they 
reported ever being arrested, found guilty of a status offense, convicted of a crime, or spent 
time in a juvenile or adult detention facility, or if they were on probation or parole at the time of 
entering YC. Analyses on the full sample include court-involved youth, non-court-involved 
youth, and youth with missing information on court involvement. Twenty-seven sample 
members were missing information on court involvement. 


4Statistical significance is estimated using chi-squared difference tests to compare differences between 
the court-involved youth and the not court-involved youth. *p < 0.1; ** p < 0.05; *** p < 0.01. 


bA status offense is defined as a noncriminal act that is a violation of the law for minors. Examples include 
running away from home and underage use of alcohol. 


E. Youth ChalleNGe program completion 


Based on the aggregate performance reports grantees submitted to DOL, court-involved youth 
appear to complete the YC portion of the program at a rate similar to YC participants overall. 
Fifty-nine percent of all YC participants completed the program. For court-involved youth, the 
completion rate was 63 percent.’ The program-specific completion rates are included in the 
program profiles (see Appendix A). The similar completion rates of court-involved youth and all 
YC youth are consistent with impressions from staff that this indicator of court-involvement was 
not a defining characteristic for youth participants. 


7 These estimates of YC completion are based on aggregate performance reports that the grantees submitted to DOL. 
These reports include the overall YC program completion rate and the program completion rate for court-involved 
participants. The progress reports do not present completion rates for non-court-involved participants. 
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lll. JOB CHALLENGE RECRUITMENT AND PARTICIPANT 
CHARACTERISTICS 


Staff from JC began recruiting youth as early as at YC enrollment and persisted throughout the 
course of the YC program. As a result, YC youth heard about JC multiple times from different 
individuals, including YC staff, JC staff, and JC participants (former YC cadets). Overall, the 
three JC programs recruited and enrolled 905 youth between January 2016 and December 2018, 
including 333 in Georgia, 301 in Michigan, and 271 in South Carolina, which exceeded the 
overall goal of 900 participants. This chapter discusses the processes programs used to recruit 
and screen youth, and describes the characteristics of JC youth, compares JC youth with youth 
who were only in YC. The findings in this chapter are based on information shared by YC and 
JC program staff during interviews, youth focus groups, and data collected on youth through the 
background information form and the follow-up survey. 


Key Findings 
e Grantees used a range of strategies to recruit YC youth into JC. Methods included starting 
recruitment early, such as at YC enrollment, maintaining outreach efforts to potential JC recruits 
over the course of YC, and engaging families to encourage youth to participate in JC. 


e Partnering with a large YC site and recruiting from multiple YC programs was beneficial for 
JC recruitment. The Georgia grantee had a larger applicant pool, because its primary YC partner 
was large and it also recruited from two other YC programs in the state. Therefore, Georgia could 
be more selective with who it accepted into JC, while other programs had to enroll most JC 
applicants to meet their enrollment targets. 


e JC youth resembled YC youth but were less likely to be court-involved. According to data 
collected through the background information form, about 36 percent of JC youth were court- 
involved relative to 41 percent of YC youth overall and 45 percent of YC youth who did not 
participate in JC. Otherwise, the groups were demographically similar. 


A. Recruitment, application, and selection 


JC programs were limited to enrolling youth from among the pool of YC graduates. Each aimed 
to enroll at least 50 youth per cohort, with the goal of at least 50 percent having prior court 
involvement. 


1. Recruitment 


To meet the recruitment goals, site staff reported using a mix of recruitment strategies including: 


e Incorporating JC as part of YC marketing. Staff hoped that the occupational training 
opportunities provided through JC would increase interest in YC. Thus, they provided 
information about JC services to prospective YC participants. 
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e Frequent JC staff outreach throughout YC. Once youth were enrolled in YC, staff started 
JC recruitment early and continued outreach efforts throughout YC. This process included JC 
staff giving presentations and having targeted conversations with individual YC participants. 
Consequently, YC reported hearing about JC multiple times from different individuals, 
including YC staff, JC staff, and JC youth. 


e Engaging families. JC staff members spoke to YC participants and their families during 
events such as YC family days or mentor visitation and through letters sent home to parents. 
Staff described the program and notified parents that their children were being considered for 
JC in the hopes that parents would encourage their children to enroll. They also aimed to 
engage and gain support from parents who expressed concern about their children being 
away from home for a second residential program. 


e Targeted outreach to court-involved youth. To meet the enrollment target for court- 
involved youth, JC program staff would identify YC participants with court involvement and 
encourage them to apply. Grantees also included additional screening questions about court 
involvement in the JC application material if prior involvement was not detected through the 
YC intake process. 


e Peer recruitment using current JC youth. Staff reported that current JC participants were 
asked to share their experiences in JC with potential JC participants, especially at recruitment 
events. This strategy was more feasible in Georgia and Michigan where the YC and JC 
programs were closer geographically. Having JC youth involved in recruitment served the 
dual purpose of marketing the program by highlighting exemplary participants and providing 
a leadership opportunity for these youth. 


e Maintaining connection between JC staff and YC post-residential staff. Although most 
JC youth enrolled in the cohort immediately following YC graduation, youth were able to 
enroll in later JC cohorts. JC staff maintained close ties with the YC post-residential staff so 
that JC could be suggested as an option for youth who were not satisfied with their post- YC 
activities. 


2. Application 


All three JC programs developed a process for current and former YC participants to apply. The 
application process varied by program but could include assessments required by community 
college partners, recommendations from YC staff, participant interviews, and disciplinary case 
reviews (see Exhibit III.1). 
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Exhibit Ill.1. JC application process steps 
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3. Selection 


JC staff selected applicants based on many considerations, including: 


e Academic proficiency tests. The college training partners established minimum test scores 
that varied by occupational track. YC youth interested in JC took placement tests, and the 
scores affected eligibility for the program. 


e Demonstrated success at YC. Programs also selected youth who had demonstrated an 
ability to succeed in the YC environment. Metrics of success included having a 
recommendation from YC staff, participating in an interview with JC staff, and not having 
any disciplinary issues while in YC. Exceptions to these requirements were considered on a 
case-by-case basis. 


e Priority for court-involved youth. Programs ave ieltneone amen ounite 


focused on meeting their DOL goal of enrolling 50 when [court-involvement] will not 
percent court-involved youth. JC program staff used be a bad thing.” 
information obtained during YC and elicited —JC staff 


additional relevant information through JC 
application questions and interviews. 


e Constraints in the gender allocation of housing. In South Carolina and Georgia, the 
housing facilities and associated supervision requirements placed some constraints on the 
number of males and females who could be accommodated. In South Carolina, this was 
rarely a binding constraint, due to the low levels of applicants of each gender. Michigan was 
less constrained, because JC youth lived in single rooms. 


Although all three grantees collected JC application materials, Georgia was the only program 
that had an applicant pool large enough that it could be selective about who it admitted into the 
JC program. Georgia’s YC program graduated twice as many youths as the programs in 
Michigan or South Carolina. Georgia also admitted youth from two other YC programs in 
Georgia. In Michigan and South Carolina, the number of interested and eligible YC youth by 
cohort rarely exceeded the number of JC slots. As a result, they accepted nearly all interested 
youth meeting a minimum standard for academic proficiency and demonstrated success at YC. 
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Conversations with staff and youth focus groups pointed to a variety of reasons why YC youth 
did not participate in JC. Some youth were deemed ineligible based on prior academic 
performance or disciplinary issues during YC. Others chose not to apply to JC. According to 
staff members, common reasons YC youth did not apply or enroll in JC included that they had 
found work; were not interested in the training provided by JC; did not want to remain in a 
restricted, supervised environment; did not want to spend four to five months in another 
program; did not like the location of JC; or were going to enroll in the military or college. 


B. Challenges in tracking Job ChalleNGe data 


Unlike YC programs, which were required to use the National Cadet Tracking System, JC had 
no unified system for tracking participant characteristics, the services which they received 
through JC, and participant post-program outcomes. This resulted in each JC site maintaining its 
own records, which varied substantially in format and comprehensiveness. Michigan used a 
system of data collection adapted from the National Guard Cadet Tracking system to add some 
JC-specific fields. Although this system included data on participant characteristics, the fields 
YC collected did not directly match the services and nature of the JC program. Both Georgia and 
South Carolina collected only limited information on JC participants. Georgia maintained a list 
of all cadets, with withdrawal dates and whether youth enrolled in occupational training courses. 
South Carolina did not have consistent data collection across cohorts and collected only the 
information for operating the JC program, such as youth rosters and occupational training 
program preferences. All three JC programs maintained their data collection independently and 
did not work in cooperation with YC programs to ensure that data could be linked between YC 
and JC. As a result of these challenges, program data could not be used to assess participant 
experiences. 


A stronger, more unified data system, that was uniform across JC grantees and linked data to YC 
data systems would have benefited the evaluation. YC staff members at each site reported 
difficulty managing and understanding the data collected for the JC program. JC staff also 
reported receiving little guidance on how to collect data or best practices for maintaining data 
and reporting outcomes. Staff further reported having limited capacity for data collection and 
hence they prioritized other activities, such as working with youth, over data tracking. 


C. Snapshot of Job ChalleNGe participants 


Based on the recruitment and enrollment practices outlined above, JC programs enrolled a group 
of participants that, in many ways, resembled the broader population of YC. For a snapshot of 
the characteristics of JC participants cohorts, Exhibit III.2. 
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Exhibit lll.2. JC participant snapshot 


Participant Snapshots: 
Who Enrolled in Job ChalleNGe? 


Characteristics of Job ChalleNGe Participants Before Entering 
Youth ChalleNGe Program 


Age Race/ethnicity 


Non-Hispanic, Black 


39% Non-Hispanic 


sCRV(-5-] eM) [o Meola olUlate (og 


ones 


4 io ye 


i 
Court-involvement and Prior employment 
school disciplinary issues 
— cae) 


30% ® employed directly before 


36% ® have any court involvement starting YC 
74% ® suspended from school 46% ® ever had a paying job for at least 
3 months 
Other statistics 
Z£orrze 
Sa Y) © 
“” 68% free “~ <1% “ <1% “~ 21% “~ 5% “~1% 
or reduced- married have a ever received unstable foster care 
price lunch child special housing involvement 
education 
services 


Source: Background information form collected from 304 youth in the three participating Job ChalleNGe (JC) programs located in Georgia, 
Michigan, and South Carolina. Analysis includes youth who enrolled in JC between July 2017 and July 2018 and who filled out a background 
information form. Youth were categorized as having court involvement if at the time of entering YC they reported ever being arrested, 
found guilty of a status offense, convicted of a crime, spent time in a juvenile or adult detention facility, or were on probation or parole. 
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Based on background information forms collected for Cohorts 4—6, 77 percent of the JC 
participants were male, and half were age 16 when they enrolled in YC. Eight percent were 
Hispanic, 43 percent were black, and 39 percent were white. Approximately two-thirds of JC 
participants reported eligibility for free or reduced-price lunch in the two years prior to YC 
enrollment, and 21 percent had received special education services at some point during their 
education. Based on the youth self-reports, 36 percent of 


“The freedom part gets me— youth had court-involvement prior to YC enrollment, lower 
with that, I'll be late [to work]. | than the target rate of 50 percent.® 
like [Job ChalleNGe] putting me 
on top of my game.” In the focus group discussions, youth reported that they 


—JC youth 4 enrolled in JC for the additional opportunities the program 
provided, including extra time to finish their GEDs, free 
occupational training, the ability to earn college credits, and free housing. Youth also reported 
that they expected the program to have a more relaxed disciplinary structure than YC. For 
example, in one focus group discussion, youth remarked that they anticipated more freedom in 
JC programs, including no uniforms and greater access to cell phones. 


Although there were many similarities between the 
characteristics of JC youth and the characteristics of 
YC youth who did not enroll in JC, there were “...when | didn’t get my GED [at 
important differences (see Exhibit III.3). JC youth YC]... | was like | have no choice 
were 5 percentage points less likely to have children. ILD eae A ent 


5 home, I’m not going to get it. So, | 
A lower proportion of black youth chose to pursue JC was like | might as well go to this 


compared to other racial groups, especially relative to program. That’s what really made 
white youth. JC youth were also statistically me come to this program.” 
significantly less likely to have received free or —JC youth 


reduced-price lunches before enrolling in YC, 
suggesting that JC youth came from families with less 
financial hardship. 


According to the performance report data that grantees submitted to DOL, across all grantees and JC cohorts, 44 
percent of JC participants were court-involved. The difference between the estimates obtained from the 
background information forms and the performance report data may be driven by the inclusion of Cohort 1-3 in 
the performance report data; the data challenges noted above made it impossible to determine the cause of these 
differences. 
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Exhibit Ill.3. Baseline characteristics of YC youth overall and by JC participation 
(reported in percentages) 


Characteristics YC only p-value? 
Age 7 
16 53 54 50 
17 36 36 34 
18 12 10 16 
Male 80 82 77 : 
Race and ethnicity site 
Hispanic 8 8 8 
Non-Hispanic, black 50 53 43 
Non-Hispanic, white 32 29 39 
Non-Hispanic, other race 10 9 10 
Foster care involvement® 1 1 1 
Free and reduced-price lunch status® 73 75 68 BY 
Unstable housing® 4 4 5 
Ever received special education services 19 19 21 
Married 
Has a child 3 5 0 on 


Educational attainment 


Last grade completed in school 


8th grade or below 8 8 6 
9th grade 26 27 23 
10th grade 40 39 42 
11th grade 22 21 24 
12th grade 


High school diploma or GED 


Employment 


Employed directly before YC 28 28 30 

Ever had a paying job for 2 3 months 44 43 46 

Sample size 984 680 304 

Source: Background information form weighted data. 

Note: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 


aWe conducted chi-squared difference tests to compare differences between the youth in YC only and JC 
youth. * p < 0.1; ** p< 0.05; *** p< 0.01. 


‘Foster care and housing status self-reported at the time of the background information form collection. 


‘Free and reduced-price lunch status self-reported based on the two years prior to background 
information form collection. 
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Despite the targeted recruitment efforts of JC staff, JC youth were seven percentage points less 
likely to have court involvement than youth who participated in YC only. Exhibit [1.4 shows 
consistent differences in self-reported delinquent behavior and justice system involvement across 
a range of measures. JC youth were significantly less likely to have ever been suspended from 
school, used marijuana in the previous six months, had a status offense such as truancy or 
underage use of alcohol, and had a prior detention in a juvenile facility. These observed 
differences could reflect less interest in the JC program among court-involved youth or could 
indicate differences in the likelihood of meeting JC eligibility requirements or being selected to 
be in the program. 


Exhibit Ill.4. Self-reported delinquent behavior and justice system involvement of YC 
youth at the time of enrollment overall and by JC participation (reported in percentages) 


Be} F-)) YC only JC p-value? 
Ever suspended 80 82 74 siete 
Used marijuana in past six months 54 58 44 ta 
Used another drug in past six months 23 24 20 
Any court involvement 41 43 36 = 
Ever arrested or taken into custody 29 30 28 
Any status offense 19 21 15 iz 
Ever convicted 18 20 15 - 
Ever detained in a juvenile facility 17 19 11 rae 
Ever detained in an adult facility 5 5 6 
On probation or parole at YC entrance 14 14 12 
Sample size 984 680 304 


Source: Background information form weighted data 


Notes: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 
Youth were categorized as having court involvement if at baseline they reported ever being 
arrested, found guilty of a status offense, convicted of a crime, or spent time in a juvenile or 
adult detention facility, or if they were on probation or parole at the time of entering YC. 


aWe conducted chi-squared difference tests to compare differences between the youth in YC only and JC 
youth. * p < 0.1; ** p< 0.05; *** p< 0.01. 


bA status offense is defined as a noncriminal act that is a violation of the law for minors. Examples include 
running away from home and underage use of alcohol. 
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IV. JOB CHALLENGE RESIDENTIAL CULTURE AND 
PROGRAM STAFF 


A distinctive feature of the JC program was its residential nature; youth lived away from home in 
a structured, quasi-military environment. However, whereas YC operated like a military boot 
camp, grantees intended for JC to have more relaxed rules and regulations. JC used elements of 
military staffing and disciplinary approaches but also introduced more freedom, space, and 
privacy to help prepare youth for independence after the program. This chapter discusses the 
housing, staffing, and disciplinary environment that defined the JC residential culture and how 
these approaches evolved over time. The findings in this chapters are based primarily on 
information shared by JC program staff during interviews, youth focus groups, and data collected 
on youth through the follow-up survey. 


Key Findings 


e The three JC programs used different housing environments, ranging from a military base toa 
rural retreat center. Despite these differences, when selecting the location, administrators from all 
three JC programs considered distances to local college partners and the YC program, as well as the 
availability, quality, and cost of the residential space. 


e According to JC administrators, the ideal staffing model for JC involved a lower staff-to-youth 
ratio than in YC but more than was originally anticipated. All three JC programs used a quasi- 
military staffing structure to provide support and discipline to youth. However, administrators realized 
that JC youth required more intensive supervision than they had initially expected, and thus reported 
that the number of staff members funded by the grant was insufficient to meet needs. 


e The JC disciplinary approach evolved from one of structured independence to earned 
freedom. Although all three JC programs initially gave youth significantly more freedom than they had 
in YC, staff determined that many youth did not yet have the self-discipline to meet their program 
goals. As such, administrators decided to tighten the disciplinary model, though it remained more 
relaxed than in YC. 


A. Job ChalleNGe housing 


The three JC programs used three unique housing models with different types of facilities, 
environments, and proximity to their associated YC partners (Exhibit IV.1). Although the three 
JC programs took different approaches to housing, Georgia and Michigan shared a similar 
model. Both were located on military facilities near their YC partner programs (Georgia’s YC 
and JC programs were co-located and the Michigan YC program was a short drive away from 
JC). The close proximity to YC meant that these JC programs could leverage base resources, 
such as dining halls, gyms, and outdoor recreational activities. Meanwhile, administrators in 
South Carolina located JC near its technical college partner but this was more than an hour away 
from YC. Although this location had some advantages, including that it was at a retreat center 
with a non-military environment and it was conveniently located near the partner college, 
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program staff reported that the distance from the YC partner’s location created logistical 
challenges. 


Exhibit IV.1. JC residential locations by program 


Fort Stewart, Fort Custer, Aiken, 
CT-Yo} ge TF} ited alte FT) South Carolina 


2, 


Type of facility Army base National Guard Retreat center run by 
training facility Clemson University 


Proximity to YC Co-located with YC YC afew miles away YC about 70 miles 


away 
Overall Base-like Base-like Quiet and rural 
atmosphere 
Type of dorm Barracks with slightly | Dorm-like Cabins with bunks 
more privacy than YC accommodations 
in the base hotel 
Access to Ability to use military Ability to use facility Ability to use retreat 
other facilities resources like gym resources, some center resources, 
and dining hall, some | shared space with such as the lake and 
shared space with military personnel a zip line 


military personnel 


Administrators reported considering the following four factors when selecting a JC location: 


Proximity to program partners. JC needed to be located close to its partners, especially the 
community or technical college where youth took classes, because youth spent a great deal of 
time there. While administrators tried to locate colleges and residential locations near one 
another, youth from all three programs had to be transported (by van) from their residential 
site to their classes. This added unavoidable costs and logistical concerns. 


Proximity to YC. Although JC youth did not regularly engage with the YC program, staff 
from both YC and JC programs coordinated around staffing and service delivery, and JC 
programs leveraged YC staff members when possible. The substantial distance between JC 
and YC in South Carolina made this type of coordination more challenging. For example, the 
JC program was less able to use YC staff to fill in for JC staff when necessary. South 
Carolina JC staff also noted that some parents did not allow their children to apply to JC 
given difficulties obtaining transportation to its rural location. 
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e Type of space. Housing varied across JC programs. Georgia’s JC program housed 
participants in barracks similar to YC housing (though slightly more private). In contrast, 
South Carolina’s JC youth were housed in cabins, and Michigan’s JC program housed 
participants in an on-base hotel for out-of-town guests. Although plain and dorm-like, 
Michigan’s accommodations had more amenities than most barracks housing. 


e Cost. All three grantees struggled with the high cost of running a residential program. 
Locating the JC program near the YC program led to cost savings for Michigan and Georgia, 
because they were able to use YC facilities and resources. For example, JC grantees had 
access to more affordable facility leases and to accommodations such as military gym 
facilities. Nevertheless, administrators at all three programs noted that feeding JC 
participants was expensive. For a variety of logistical reasons, JC cadets did not eat the same 
food as YC participants, and their eating arrangements generally cost more than in YC, 
including in Georgia where YC and JC were collocated. At the Georgia JC location, for 
example, JC youth had to be bussed to a cafeteria on the base, where food was more 
expensive than what YC youth ate. Similarly, in Michigan, JC youth were fed through the 
hotel where students lived, which was more costly than feeding YC youth. 


Regardless of the challenges these location-based factors presented to administrators, youth in 
focus groups noted that JC housing was clearly preferred to YC housing. They indicated that JC 
housing was more spacious, had more privacy, included more access to outdoor activities, and 
provided more opportunities for interaction with nonparticipants. They also indicated that the 
food was better. Even in Georgia, where JC was located on the same base as YC, JC youth still 
had more space and amenities in recognition of their status as YC graduates. As one youth 
explained, “You go from sharing a room with 40 people [in YC], to your own room and sharing 
a bathroom [in JC].” 


B. Staffing Job ChalleNGe to meet participants’ needs 


JC staff members implemented and managed the program and provided support and discipline 
for the youth. JC staff were not responsible for delivering occupational training, because it was 
provided by programs’ community and technical college partners. Findings regarding staffing 
are: 


e All three JC programs had well-defined staffing structures with clear roles for each job. 
At the highest level, some staff members were responsible for designing and monitoring JC, 
while other staff members worked directly with youth to ensure their safety and success. 
Each JC program had a senior administrator who oversaw both YC and JC, a JC-specific 
project coordinator for day-to-day management, and a small number of residential advisors. 
JC programs also had other unique staff roles, such as a lead counselor to provide additional 
oversight for the staff and a JC-specific academic counselor. Overall, the JC programs had 
fewer dedicated staff members than did their YC partners. YC staff typically included these 
same roles with larger numbers of staff, along with recruiters, kitchen staff, IT staff, and 
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others in support roles. JC programs had to forgo providing these additional support services 
unless they were able to find ways to leverage the YC support staff to serve JC participants. 


e Faculty and administrators at partner community colleges played a key role in the JC 
program. Because JC youth spent substantial portions of their day attending occupational 
skills training at a community or technical college, instructors at these institutions were key 
to training and disciplining youth. Over the course of the JC grant, Georgia and South 
Carolina switched from using adjuncts to full-time faculty when they realized that the 
classroom management and instructional style of full-time faculty better supported youth 
learning. Aiken Technical College in South Carolina also hired a part-time administrator to 
serve as a JC liaison. This administrator attempted to ensure that students’ needs were being 
met, handling functions like 
class scheduling and 
communication between JC 
and college staff. Because 
college instructors were not 


Implementation tip: 
Train and leverage YC staff 


All three sites found it helpful to leverage YC staff 


always familiar with JC and 
youths’ education 
backgrounds, JC staff 
members recommended 
orienting them with JC before 
the start of the semester. 


JC administrators needed 


members to provide sufficient levels of oversight and 
support. This process involved hiring staff who had 
experience working with the youth at YC and 
leveraging existing YC staff members. For example, 
Michigan used YC staff as backup for JC staff on 
their days off. This approach was more feasible in 
Georgia and Michigan, because those JC residential 
environments were near their YC partner. 


However, there were also challenges in sharing staff 
across YC and JC. As both youth focus groups and 
staff interviews indicated, substitute YC staff 
members were not always aware of the different 
disciplinary policies for JC and other differences in 
the way JC operated. Additional training for YC staff 
on the disciplinary rules and culture of JC would have 
helped to address this challenge. 


more staff than expected to 
implement JC. 
Administrators from all three 
JC programs either hired or 
reported wanting to hire 
additional JC staff. Although 
JC administrators initially 
expected that JC youth would 
require substantially lower 
levels of supervision than those in YC, over time, they decided that youth needed more 
supervision and support. Staff felt that, after youth had gotten used to the highly structured 
environment of YC, they did not yet have the self-discipline needed to conduct themselves 
appropriately on their own. For example, they indicated that youth needed to be engaged in 
supervised activities when not in class. Administrators from all three programs stressed that 
JC funding was insufficient to provide the level of staffing they eventually believed was 
necessary. Staff described that this factor sometimes limited their ability to provide one-on- 
one counseling or to break youth into smaller groups for study sessions or outdoor activities. 
To compensate, JC program administrators leveraged YC staff members when possible. JC 
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administrators also explained that JC staff members would play multiple roles, such as 
providing both counseling and conducting IT updates, to cover all program needs. 


e Generally, youth reported feeling that JC staff members were helpful and that they 


could talk to staff about any concerns. In focus groups, youth expressed their gratitude for 


JC staff, especially the resident advisors. One participant from Georgia described them as 
there to protect and guide them: “Some people break down and need guidance, and they 
help.” 


Exhibit IlV.2. Youth perspectives: Quality of staff help 


JC participants generally had positive opinions of staff, especially for help provided during the 
program. They thought, however, that staff were less helpful after they left the program. Most 
youth indicated that JC staff understood their needs, helped them learn, and helped them solve 
problems. Lower percentages of participants thought that the staff were good or very good at 
keeping in touch with them or providing help after the participants left JC. 


60% 


40% ™ 76 percent of 
Ve jood 
ic. youth reported 
that they 
20% : 
a 31 believed at least 
one staff 
0% member really 
Helps you Understands Helps Help after Attempts cared about 
learn, your needs you solve leaving JC to keep 
academically problems in contact itatelaamrslale| they 
or with you ; 
eine e staff member 


F-) ofolU| m e= 1610) aT I 
Notes: Analysis includes all youth in JC Cohorts 4-6 who filled out the follow-up survey. Youth things 

were asked to rate the quality of staffs help on a four-point scale (1 = poor, 2 = okay, 
3 = good, 4 = very good). 


Source: Follow-up survey weighted data. 


C. An evolving approach to discipline 


Like YC, JC adopted aspects of a military-style discipline structure and upheld a series of rules 
governing many components of program life, such as dress, study time, cell phone use, and 


contact with the outside world. However, staff members also recognized that it was important to 


provide youth with more freedom than they had in YC. As mentioned earlier, JC staff believed 
that JC was meant to help prepare youth to make healthy choices after leaving the residential 
program environment and entering school or work. 
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1. Initial tone of “structured independence” 


Mathematica 


In the early round of site visits, program staff described their attempt to create a disciplinary 


structure that was strict yet gave youth 
flexibility to make decisions on their own. 
One JC administrator described this 
approach as “structured independence,” with 
an atmosphere that was more academic than 
militaristic. Another JC administrator 
elaborated on this approach, explaining that 
the goal was to give youth, “the 
opportunity...to fail or succeed through their 
own choice, even if we guide them.” 
Although youth still had boundaries and 
oversight, they had more freedom than they 
did in YC. For example, youth in all three 
JC programs could have cell phones and 
leave on weekends to visit their families. In 
the Michigan JC program, youth did not 
have to wear uniforms, and in the other two 


Implementation tip: 
Create a written manual or 
handbook for program rules and 
disciplinary procedures 


Some JC youth complained during focus 
groups discussions that program rules 
changed over time or were implemented 
inconsistently. At times, YC staff, who 
lacked familiarity with JC rules, provided 
temporary help in JC programs. Written 
discipline manuals (which all JC 
programs eventually created) could 
improve clarity for both youth and staff 
members, but their content must be 
followed and kept up to date. 


programs, they wore uniforms only at certain times. Infractions were generally punishable by 
having privileges taken away (for example, not being able to leave on the weekend). 


2. Move toward increased discipline or “earned freedom” 


Community and technical college 
discipline 


structures. For the most part, youth were 


of assignments. Individual instructors could 
choose to discipline youth or refer offenses to 
JC. Staff or resident advisors also provided 
additional support as needed. In the South 
Carolina JC program, for example, staff 
typically waited outside the classroom door to 
deal with classroom disruptions as needed. 


Because youth spent significant amounts of 
time at partner community and technical college 
campuses, these sites also needed disciplinary 


subject to regular college rules and disciplinary 
procedures, such as for the timely submission 


Over time, staff members across all three 
JC programs increased the level of 
oversight and rules. They felt that their 
early approach of structured 
independence was challenging for youth, 
who were perceived to not yet have the 
self-discipline or maturity needed to 
succeed in their college-level classes 
without more rules and support. One staff 
member explained that youth still needed 
guidance: “It was really premature on our 
part to think them ready to be college 
students.” Another said that, initially, 
they had wanted to implement a culture 
of “if you want to be treated like an adult, 
act like an adult.” JC administrators 
described moving to a model that one 


administrator referred to as “earned freedom.” Youth had to demonstrate their readiness for 
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privileges. For example, administrators from Georgia and South Carolina decided to allow cell 
phones only for youth without disciplinary issues after several weeks, and even then, cell phone 
use at all programs was eventually limited to certain times. Michigan JC added a two-week 
acclimation period with stricter rules to help youth get used to life in JC. The South Carolina JC 
program also added more staff for increased supervision and to enforce additional rules. 
Nevertheless, despite the additional supervision and structure, the disciplinary environment was 
still less strict than in YC. 


Youth perspectives: On discipline 


Overall, youth found the JC discipline structure to be less severe and more supportive than what they had 
experienced in YC. One Georgia youth explained: 


“JC is more like ‘we’re here to guide you in the right direction.’ YC is more ‘put 
our foot in your back and make you go the right direction.’” —Georgia participant 


However, despite appreciating this relative leniency, youth also complained either that staff were too 
inflexible in the way they conducted discipline, or conversely, punishments seemed to be given 
inconsistently, especially compared with YC. Youth were especially concerned about cell phone policies 
and how they had changed over time. A focus group participant explained that he and his peers were very 
motivated to follow the rules so that they would earn cell phone privileges, and then were especially upset 
when those privileges were revoked temporarily due to someone else’s behavior. 


3. Infractions and punishments 


JC staff members reported that although some youth committed infractions during program 
participation, most were not very serious. Respondents highlighted some of the most common 
infractions, including uniform violations, cell phones use during times when they were not 
allowed, and challenging authority. More serious infractions included fighting, positive drug 
tests, trespassing to other parts of the base, and smoking. 


As described above, punishments for infractions, especially those that were not serious, generally 
involved taking away privileges or failing to earn them in the first place. Staff members said that 
they also tried to relate discipline to the work world. For example, a staff member said he would 
ask youth, “What would happen in the real world, if you did this at a job site?” Another JC 
program brought back a former cadet who had been fired from his job to tell his story. More 
serious infractions were brought to the attention of program leadership. Georgia JC also created a 
youth board that tried infractions like a court; youth who held a leadership position on this board 
determined appropriate punishments or solutions for their peers. 
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V. JOB CHALLENGE PROGRAM SERVICES 


With the goal of improving youth employment outcomes, JC grantees were required to 
implement activities and services in four areas: (1) occupational skills training; (2) individualized 
career counseling and academic counseling; (3) work-based learning and exposure to the work 
world; and (4) leadership development activities that encourage responsibility, employability, 
and other positive social behaviors. Service delivery in JC focused on the implementation of an 
intensive occupational skills training program. Beyond the required services, grantees also 
offered access to education services, including secondary education services for youth who had 
not completed their credential during YC and general postsecondary courses for those who 
qualified. 


In this chapter, we discuss the services provided through JC and JC program completion. These 
findings are drawn from several data sources including site visits, the follow-up survey, and 
grantee performance reports. The chapter also includes call-out boxes containing youth 
perspectives about the quality of specific services and of the JC program overall. 


Key Findings 


e Job ChalleNGe provided access to intensive occupational training and supplementary 
education. JC programs enrolled youth in community colleges where they took general education 
courses and participated in certificate-based vocational training programs. Youth who lacked a high 
school diploma or GED also participated in secondary education services. All youth received some 
education service or training. 


e Partnerships with community colleges were key to providing occupational training programs. 
JC programs formed partnerships with community colleges to provide occupational training and other 
related program services. All participants received some occupational training, and 87 percent 
received an industry-recognized certification. 


e Court- and non-court-involved youth received largely the same services. JC staff members did 
not differentiate the services they provided to youth based on whether the youth were court-involved. 
The only exception was that staff steered some youth with criminal backgrounds away from 
occupational training in fields in which it might be difficult for people with criminal records to obtain 
employment. 


A. Overview of Job ChalleNGe services 


The overarching goal of the JC program was to prepare youth for future employment by 
providing a comprehensive set of training and support services. Youth in JC reported high levels 
of participation in all types of services, as shown in Exhibit V.1. All of the youth surveyed 
reported receipt of occupational training. Most youth also received a range of other services and 
accreditations, including occupational training credentials (73 percent), industry-recognized 
certifications (87 percent), education services (97 percent), job-readiness services (75 percent), 
and other support services (91 percent). The rates of receipt of overall services and accreditations 
were very similar across sites, despite that the details of service delivery varied. Although our 
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data on service receipt is drawn from the follow-up survey that we conducted with the final three 
cohorts of JC, information collected during site visits suggests that service delivery was 
consistent across the cohorts. 


Exhibit V.1. JC services and credentialing (reported in percentages) 


100% 100100100100 97 97 
90 93 
75% 73-73-14, 75 
Bi rota 
south Carolina 
0, 
50% [michigan 
Wi ceorgia 
25% 
0% 
Occupational Occupational Industry Educational Job Other 
training training recognized services readiness support 
credential —_ certification training services 


Source: Follow-up survey weighted data. N = 150. 
Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up survey. 


For the most part, there was no distinction in service provision between court-involved and non- 
court-involved youth. Staff did not consider court involvement in deciding what services youth 
would be offered, and in many cases, they did not even know which youth were court-involved. 
One exception to this finding was assignment to occupational training programs. Staff expressed 
concern that a criminal record—at least for certain types of offenses—could limit the 
opportunities for cadets in certain fields, primarily health care. As such, academic and career 
counseling staff members generally encouraged cadets with criminal backgrounds to pursue 
training in nonmedical fields. 


B. Education and occupational skills training 


The primary goal of adding a JC extension to the YC program was to “expand and enhance the 
[YC] program’s job skills component’”’ to put participants on a path to employment. To 
accomplish this goal, the JC program was structured around vocational training programs 
designed to equip participants with the skills and credentials needed for employment in a given 


7s. Department of Labor, “Notice of Availability of Funds and Funding Opportunity Announcement for National 
Guard Youth Challenge and Job Challenge Program.” Available at https://www.doleta.gov/grants/pdf/FOA-ETA- 
15-01 .pdf. 
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field. All three JC programs partnered with community colleges to develop occupational training 
programs and a general JC curriculum. 


Although the programs differed in many 
JC community college partners ways, they all took similar steps to establish 
Georgia these partnerships and the JC occupational 


Institution: Savannah Technical College training programs. 
Location: Savannah, Georgia 


e JC programs collaborated with 
Institution: Kellogg Community College community college partners to decide on 
(Regional Manufacturing and Training Center) the content and structure of college 


Location: Battle Creek, Michigan 
: courses for JC participants. These 
South Carolina : ; - 5 
collaborations included determining which 


Institution: Aiken Technical College ; ; ae 
Location: Aiken, South Carolina occupational skills training courses to offer. 


Programs based these decisions on local 
labor market demand and the interests and 
academic skills of the youth. JC youth typically enrolled in dedicated sections of the courses, 
without other college students. Staff and college partners modified the length, structure, and 
timing of the courses to better fit within the JC program time frame. One JC staff member 
noted how the program’s college partner had really taken ownership of the program and, 
when the program handed over its youth to 
college faculty and staff members, it knew 
that those staff viewed the youth as their 
students. This sentiment was echoed across 
the other two programs, with one staff 
member noting how the program’s college 


Secondary education services 


Although the JC program was 
designed to give youth postsecondary 
education and training, 60 percent of 


partner had “bent over backward” to cadets had not completed their 
support and accommodate JC. secondary education (a diploma or 
high school equivalency certificate) by 
e Youth enrolled as college students at the start of JC. JC programs therefore 
community college partners. As college needed to offer secondary education 
students, youth were able to take general services to participants. Approximately, 


education courses, such as math or English Ba pete eel Sue ved youu 
reported taking courses in GED 


courses, as well as college courses focused preparation or to prepare for a high 
on things like study skills or how to school diploma. 
navigate the college environment. As part 
of JC, youth had full access to campus 
facilities and events; learned to interact with 
students, instructors, and faculty; and became aware of and utilized college services, such as 
tutoring and career counseling. Across the three sites, 65 percent of participants reported 
taking courses for college credit while at JC, as shown in Exhibit V.2. According to JC staff 
members, by the time some youth completed JC, not only had they earned a certificate or 
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credential, they had also completed about one-third to about one-half of the credits needed 
for an associate degree. '° 


e Programs were able to leverage financial resources through partner community 
colleges. A key benefit to partnerships with the community and technical colleges was 
support in helping to pay for training. To cover the cost of providing education, JC programs 
leveraged financial aid available to partner community college students.!! All three JC 
programs were also able to use Pell grants and partner with college staff who guided both 
youth and JC staff members through the financial aid application process. 


e JC staff worked with community college partners to provide academic and career 
counseling. Counseling was designed to help JC participants plan their long-term education 
and career trajectories. Counselors used sessions with youth to determine their education, 
training, and supportive service needs and took advantage of the resources and experiences of 
college staff in providing similar services to college students. Staff felt that this support was 
particularly important for participants interested in pursuing additional education following 
JC. Across the three sites, 32 percent of participants reported receiving academic counseling, 
such as help identifying and applying to education or training options after JC. We describe 
career counseling in more detail in the section below on employment services. 


Exhibit V.2. JC participants taking courses for college credit (reported in percentages) 


65 
60% a 
40% 
20% 
0% 


Total Georgia Michigan South Carolina 


66 


Source: Follow-up survey weighted data. N = 150. 
Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up survey. 


'0 Because the data NSC collected for this project does not include information on credits received or number of 
credits required for a degree, we are unable to quantify the number of participants who made substantial progress 
toward an associate degree. 


ohh Georgia, JC participants were eligible for the state’s Hope Grants. In Michigan, the JC program was able to 
leverage an existing Trade Adjustment Assistant Community College Career Training (TAACCCT) grant. In 
South Carolina, the JC program was able to use a state lottery tuition assistance fund. 
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Benefits and drawbacks of offering JC-only occupational training programs 


Benefits: During site visits, program and college partner staff members noted important 
advantages of having classes that consisted only of JC students 


e Easier to manage discipline. These dedicated sections made it easier for program staff to 
be present and easily intervene to correct discipline issues, without disrupting the education 
experience of non-JC students. 


e Better able to tailor instruction. Staff thought the JC-only courses were better able to 
calibrate the pace of instruction based on the needs of JC students. JC youth were often 
younger than other college students and had somewhat different education backgrounds than 
non-JC students. One course instructor noted that he adjusted the pace of JC-only classes 
and often built in frequent breaks so that JC cadets could get acclimated to the college 
environment. 


Possible drawbacks: At least one JC participant expressed that being in JC-only courses led 
youth to feel set apart from other college students and therefore like they were not getting the full 
college experience. 


1. Occupational services and credentials 


Occupational skills training was a central component of the JC program. Staff matched youth to 
occupational services based on youth’s interests, course availability (i.e., not all courses were 
offered during each cohort), and the fit based on youth aptitude and background experience as 
assessed in career and academic counseling services. As highlighted above, 100 percent of 
surveyed youth reported receiving some form of occupational skills training. Exhibit V.3 shows 
the occupational training programs taken by JC participants by site. The most popular 
occupational training programs were 


nurse’s aide/certified nursing assistant VW 9. 
(CNA) and welding, although UDG SUDA IES 
enrollment varied by site. For example, Cadets in South Carolina’s Aiken Technical College 


tower technician program earned a certificate in 
basic tower and wireless installation and 
maintenance. As part of this training program, youth 


only 7 percent of Michigan participants 
received nurse’s aide/CNA training 


relative to 30 percent in Georgia and 31 learned technical and safety aspects of the job and 
percent in South Carolina. This they participated in a weekly, 80—100-foot tower 
variation was partially due to climb at an active tower for a television station in 
differences in programs offered by site south Caroling: y 
and cohort. 
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Exhibit V.3. Occupational training services in which JC participants enrolled (reported in 
percentages) 


South 


Services received Georgia Michigan Carolina 


Nh 
oO 
oO 
oO 
N 
oO 
= 


Nurse’s aide or certified nurse assistant 


—_ 
No 
oO 
—_ 
K 
to) 
[o) 


Computer networks 


—_ 
AK 
(o>) 
oO 


Automotive or electronic technician 6 


© 
oO 
[o) 


Culinary or food production worker 4 


(o) 
o 
[o) 


Electrical systems 3 


ae 
o 
[o) 


Heavy equipment operation 3 


oO 
No 
K 


Advanced manufacturing 2 


Other@ 3 


A 
nN 
oO 


Source: Follow-up survey data weighted data. 
Note: Analysis includes all youth in JC Cohorts 4-6 who completed the follow-up survey. 


aOther write-in trainings reported by youth included manual machining, phlebotomy, sports medicine, and 
digital media. Overall, most JC participants completed their job training and earned credentials. Exhibit 
V.4 shows the share of JC participants who received a credential for completing their job training 
program. Among all participants who started JC, 76 percent received a credential after the training. 
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Exhibit V.4. JC youth completing training and receiving certification (reported in 
percentages) 


76 = - i 
60% 
40% 
20% 
0% 


Total Georgia Michigan South 
Carolina 


Source: Follow-up survey data weighted data, N = 150. 
Note: Analysis includes all youth in JC Cohorts 4-6 who completed the follow-up survey 


Noncompletion of job training credentials was largely driven by JC program drop-out. Among 
participants who did not complete a credential, 75 percent did not finish the program, as shown 
in Exhibit V.6 . One possibility is that youth who experienced challenges with their job training 
program dropped 
out of the JC 
program entirely. 
However, only 5 
percent of JC 
dropouts reported 
they did not like 
their job training 
field or classes. 
Among youth who 30% 
completed the JC 


45 
22 
program but did 20% 
not receive a 
credential, the 10% 8 
most common | 
reason was that 0% 
Poor 


24 
they needed to Okay Good Very good 


complete Source: Follow-up survey data weighted data. N = 132. 
additional work, Note: Analysis includes all youth in JC Cohorts 4-6 who completed the follow-up survey. 


Exhibit V.5. Youth perspectives: Quality of job-training programs 


Most JC participants surveyed reported that they thought the job- 
training programs were of high quality. 


40% 


such as more 
courses or an 
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assessment. Ten percent of participants also said there was no credential available for the 
industry in which they wanted to work. 


Exhibit V.6. Reasons for youth not receiving a credential (reported in percentages) 


South 
Reason for not receiving a credential We) r-1| CT-Yo) fe] F: Mmmm Coanl (et: 1a mm Ox- 10) [LAT | 
| left before | completed the program 75 83 69 66 
Needed to retake one or more courses | took at JC to 13 17 9 11 
get credential 
JC did not offer a credential program for the industry | 10 17 5 0 
wanted to work in 
Needed to take a test or assessment in addition to 8 4 3 23 
coursework completed at JC to get credential 
More courses needed for credential than available 5 0 16 0 
through the JC program 
Needed additional work experience to get credential 1 0 4 0 
Sample size 36 13 16 
Source: Follow-up survey data weighted data. 
Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up survey and reported 


that they did not complete a credential. 


In addition to the occupational training programs, youth also worked toward other industry- 
recognized certifications. For example, more than three-quarters of participants received 
Occupational Safety and Health Administration (OSHA-10) certifications, as shown in Exhibit 
V.7. Other common certifications included forklift certification; cardiopulmonary resuscitation 
(CPR) certification; and WorkKeys, a career readiness certification. '? Certifications were 
designed to provide participants with a way to demonstrate to employers the job skills they 
learned and improve the employability of participants following JC. 


ie WorkKeys assessments are used by many workforce development system providers across the country to measure 
participants’ hard and soft skills and to provide employers with a standardized measure of these skills, 
demonstrated through various certificate tiers. 
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Exhibit V.7. Other credentials youth received through JC (reported in percentages) 


South 

Services received CT-Yo} ge F-] Michigan for-Txe) Tar) 
OSHA-10 (Occupational Safety and Health 76 82 85 54 
Administration) 

Forklift 37 49 14 46 
Cardiopulmonary resuscitation 32 39 12 50 
WorkKeys? 21 5 23 48 
Other certification program 3 0 6 6 
None 13 9 9 28 
Sample size 150 54 67 29 


Source: Follow-up survey data weighted data. 
Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up survey. 


@Details of the WorkKeys career readiness certification can be found at 
http://www.act.org/content/act/en/products-and-services/workkeys-for-employers/assessments.html. 


C. Employment services 


All three JC programs worked to determine the right mix of education, training, and supportive 
services needed to help each participant achieve his or her long-term education and career goals 
Employment services were clearly a central service component in which programs expected 
youth to participate (Exhibit V.8). JC programs offered participants a range of services to expose 
them to career options and provide them with the tools needed to find jobs. They also offered 
workplace experiences designed to give participants a more realistic picture of what employment 
would look like in different industries. Across the three sites, 43 percent of participants were 
exposed to work environments through field trips, job-shadowing opportunities, or internship 
experience (not presented in exhibit). The following section details the employment services that 
JC offered. 


Career counseling and job search assistance. Career counseling, offered formally through 
community college partners, was designed to help JC participants identify and achieve 
employment goals. JC staff also helped JC participants with employment support services such 
as job search assistance. Across the three sites, nearly half of surveyed participants reported 
receiving some job search assistance (46 percent), and about one-third (32 percent) reported 
receiving formal career counseling. Twenty-one percent of participants received help applying to 
another vocational training program following JC. 


Field trips to local employers and guest speakers. All three JC sites provided participants with 
opportunities to visit workplaces through field trips to local employers, such as local factories or 
kitchens. For example, in Georgia staff took youth on a trip to the Westin Hotel culinary group, 
where the head chef demonstrated pastry-making techniques. Across the three sites, 29 percent 
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of youth reported having attended at least one of these field trips. JC program staff also 
explained in interviews how they sometimes brought in guest speakers to meet with youth. For 
example, a South Carolina staff member invited a human resources representative from a local 
company to talk about the hiring process and industry representatives to help teach about 
different types of career opportunities. 


Exhibit V.8. Employment services received by JC participants (reported in percentages) 


South 
Services received Georgia WY Efed alte FeV) Carolina 
Any employment service 15 78 15 70 
Help searching for a job, including help 46 47 46 43 


filling out an application, writing a 
resume, or going on an interview 


Career counseling 32 44 23 24 
Help applying to a vocational training 21 20 21 24 


program to attend after JC, including 
help with an application or interview 


Field trips to business places/work 29 29 36 19 

environments 

Job-shadowing opportunities 23 44 7 4 

Internship experience 13 21 2 12 
None 25 22 25 30 
Sample size 150 54 67 29 


Source: Follow-up survey data weighted data. 
Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up survey. 


Job shadowing. Across the three sites, 23 
percent of youth reported participating in job 
shadowing. This activity was a larger component 
of the Georgia JC program, with 44 percent of 
participants in job shadowing, relative to 
Michigan and South Carolina, with 7 and 4 It was not always easy for JC 
percent participation, respectively. There was poate Ou Cole Je Puree 


ae : ; provide participants with work 
also variation in access to job shadowing by experiences, especially paid 


Challenges in offering work 
experiences 


occupational training program. Job shadowing employment. JC staff reported that the 
was most common in medical fields, such as main obstacle was employer liability 
CNA training, first-responder training, or patient concerns. One staff member at a JC 


care coordinator training. Staff reported that Oo uN Secueces Mamany crue 
manufacturing companies near the JC 


these experiences also occasionally led to program could not legally hire workers 
additional volunteer or even paid positions. under 18, which limited work 
opportunities for many youth. 


Internships and part-time work. Youth 
sometimes had opportunities to obtain direct 
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work experience. JC programs sometimes assisted youth in finding part-time work on or near the 
program’s residential location, even if these positions were not directly related to the field of 
training. These opportunities also varied by program, being most common in Georgia (21 
percent), less common in South Carolina (12 percent), and very rare in Michigan (2 percent). 
They were also common in medical fields. For example, some youth in Georgia and South 
Carolina obtained work experience through a rotation in a nursing facility. Staff from the 
Georgia JC program reported partnering with a local workforce system agency to provide paid 
internships for a small number of participants; these internships were typically in fields outside 
of the participants’ training area, but they did provide income and work experience. 


D. Non-academic support services 


JC programs were designed based on the theory that a holistic set of support services is needed to 
promote employment success among youth. Both formal and informal life skills training (e.g. 
budgeting, independent living, employability), leadership training, and personal support services 
helped youth build soft skills needed to succeed in the workplace and provided resources to 
improve well-being. Overall, these opportunities tended to expand on those offered in YC, and 
provided more opportunities focused on career planning and placement. Non-academic support 
services were widespread in JC programs, with 91 percent of participants reporting they received 
at least one service (Exhibit V.9). As with most services in JC, staff members did not distinguish 
between court-involved and non-court-involved individuals in service delivery. 


Exhibit V.9. Non-academic support services received by JC participants (reported in 
percentages) 


South 
Services received Be} F-)) Georgia Michigan Carolina 
Any non-academic support service 91 88 90 97 
Got help or advice from JC program staff 72 74 76 62 
Got help or advice from a mentor 51 50 58 43 
Life skills classes or training (including budgeting, 50 49 55 45 


banking and other financial skills, independent living, 
employability class, etc.) 


Communication or public speaking training 30 33 31 24 
Leadership training 48 52 50 38 
Leadership experience (for example, experience as 46 54 43 36 
a squad leader, platoon guide) 

Health services 29 37 19 28 
Mental health services 18 22 18 12 
None 9 12 10 3 
Sample size 150 54 67 29 


Source: Follow-up survey data weighted data. 
Note: Analysis includes all youth in JC Cohorts 4-6 who completed the follow-up survey. 


47 


Job ChalleNGe Final Report Mathematica 


Work readiness and life skills coursework. JC programs provided training and education in 
both hard skills (technical abilities with measurable competency) and soft skills (interpersonal 
and personality-based skills) needed to find and maintain a job (either civilian or military) or 
succeed in higher education. All three sites offered communication or public speaking training 
(30 percent of youth participated) and life skills classes or trainings (50 percent participated). 
Although the specific elements of the life skills training varied across JC programs, some 
examples of content included professional skills training, such as professional dress, using a time 
clock, etc.; information about military careers; harassment and bullying in the workplace; study 
skills classes; and financial literacy classes. Compared with similar YC services, staff reported 
that these services were longer, more career-focused, and typically offered as a course generating 
college credit. 


Leadership 
development. 
aus ; Exhibit V.10. Youth perspectives: Quality of leadership 
Leadership development Seponunttice 
occurred through both 


Most JC participants who were surveyed reported that they 


l hip traini 
eadership training and thought the leadership opportunities were of high quality. 


leadership opportunities 


within the program. ia z 
Combined, 59 percent 
of youth received some = - 
leadership development = 
service, with 48 percent 20% 
receiving leadership 5 
training, and 46 percent 10% 
taking on a leadership Ld 
role. Examples included 0% 
Poor Okay Good Very good 


serving as squad leader 
during exercise and Source: Follow-up survey data weighted data. N = 139. 

ac ; Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up survey. 
transitions, serving on a 
leadership council that 
planned social events 
and settled dispute 
between participants, 
helping other participants in study hall and tutoring, assisting instructors during classroom 
training, and representing the program through public speaking and recruitment assistance. 
Leadership opportunities were considered to be an earned privilege, although sites varied in how 
they assigned them. Two programs used a military-style concept of rank in which only youth 
who had achieved high ranks were eligible for leadership positions. Staff also tried to rotate 
participants through positions so that more participants would have an opportunity to serve in a 
leadership role. 
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Support services. JC programs also strove to provide youth with the formal and informal 
support services needed to succeed. While at JC, some participants took advantage of both 
physical (29 percent) and mental health services (18 percent). Participants also relied on program 
staff and assigned mentors for guidance during the program. Many participants reported turning 
to program staff (72 percent) and mentors (51 percent) for advice during the JC program. 


E. Post-residential services 


JC staff relied heavily on YC staff to monitor whether youth were achieving their post-program 
goals of being engaged in a productive activity, such as postsecondary education, training, 
employment, or the military. Because 82 percent of participants went straight from YC to JC, the 
JC post-program period usually overlapped with the one-year post-residential phase of YC. JC 
staff typically had far less capacity to follow up with youth in the post-residential phase of JC 
relative to YC staff, who routinely monitored youth progress in the post-residential phase. 
Typically, JC program staff members “passed back” youth to their YC counselors after the 
completion of JC, thus relying on YC staff for the post-residential phase follow-up. JC program 
staff also indicated in interviews that they typically relied on youth to be proactive in providing 
updates or used more 

passive ways to identify 

youth progress, such as Exhibit V.11. Youth perspectives: Connection with staff 
scanning social media. following JC 


More than half of participants surveyed thought staff attempts to 


The relationship keep in touch after JC were either okay or poor 


between youth and their 
mentors was designed, 
however, to endure 30% 
beyond JC. The mentor 


33 

relationship formed = = 
during YC was intended i 
to last for at least the 
full-year post- _ 
residential period of 
YC. The mentor, who 

0% 


19 
was chosen as someone 
. Poor Okay Good Very good 
with whom the youth 
had ian Al : Source: Follow-up survey data weighted data. N = 150. 
ad a relationship prior Note: Analysis includes all youth in JC Cohorts 4-6 who completed the follow-up 
to the YC program, survey. 


committed to helping 

youth make the most of 

the lessons that they learned at YC and JC. Mentors played a limited role in JC, although JC staff 
did facilitate check-ins and outings with youth and their mentors. Therefore, the post-program 
relationships with mentors largely mirrored that of YC participants. Approximately two-thirds of 
youth reported staying in touch with their mentor after JC (Exhibit V.12). 
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Exhibit V.12. Share of JC youth who reported keeping in touch with their mentors 
following JC (reported in percentages) 


0% 


69 
64 63 
60% 58 
40% 
20% 


Total Georgia Michigan South Carolina 
Source: Follow-up survey data weighted data. N = 150. 
Note: Analysis includes all youth in JC Cohorts 4-6 who completed the follow-up survey. 


F. Job ChalleNGe 
completion 


Overall JC completion 
rates, as reported in the 
grantee performance 
reports submitted to 
DOL, were high, with 89 
percent of youth 
completing JC in South 
Carolina and 93 percent 
in Georgia. The overall 
JC completion data for 
Michigan was not 
available. Completion 
rates for court-involved 
youth ranged from 43 
percent in South Carolina 
to 86 percent in Georgia. 
Completion rates for each 
program are shown in the 
program profiles in 
Appendix A. As 
described in Appendix B, 


Exhibit V.13. Youth perspectives: Overall quality of JC 
program 


Most participants rated the JC program positively, with 41 
percent of participants rating it as “very good” and 24 percent 
rating it as “good.” 


40% 


30% 


41 
23 
20% 
12 
- _ 
0% 


24 
Poor Okay Good Very good 


Source: Follow-up survey weighted data. N = 150. 


Note: Analysis includes all youth in JC Cohorts 4—6 who completed the follow-up 
survey. Participants were asked, “How would you rate the quality of your JC program 
with regards to your overall JC experience?” 
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staff reported some concerns about the consistency of the data reported in the grantee 
performance reports. The differences in the completion rates of court-involved and non-court- 
involved youth also contrasted with the perceptions of grantee staff that court-involved youth 
had similar rates of attrition to other youth. In fact, staff at two JC programs thought these youth 
might be more motivated to achieve success in the program than their peers, because they better 
understood what was at stake if they did not complete the program. 


Twenty eight youth who left the JC program completed the survey, providing a glimpse into their 
reasons for leaving. Although almost a quarter of youth did not complete JC because they were 
asked to leave, the rest of the youth who left JC did so voluntarily. Some of the most common 
reasons youth cited for leaving the program included not liking the program overall (28 percent), 
a family member becoming ill (23 percent), not liking the style of discipline in the program (15 
percent), not liking the other youth (13 percent), not liking the staff (12 percent), and being 
homesick or wanting to return home (9 percent). These responses were consistent with JC staff 
reports and responses from focus group participants that youth left because they wanted to be 
with family or friends or felt the environment was too restrictive. 
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VI. JOB CHALLENGE PARTICIPANT OUTCOMES 


The primary goal of the Job ChalleNGe program was to improve the outcomes of participants 
after the JC program through access to education and job training. As previous chapters have 
highlighted, each component of the program was designed to provide participants with the skills 
and resources to improve post-program outcomes. In this chapter, we present analyses of the 
education, employment, and military outcomes of JC participants. We also show the arrests and 
convictions of participants after leaving JC. Although we are unable to present causal evidence 
of the impact of JC on these outcomes, for context, we present a comparison of JC participants’ 
outcomes to outcomes for YC participants. The findings in this chapter are based primarily on 
data from the JC follow-up survey, NSC, and state criminal justice administrative records. 


Key Findings 


e Across the three sites, 86 percent of JC participants were involved in a productive activity 
approximately 14 months after JC and most avoided criminal justice involvement. This is driven 
by the fact that 81 percent of JC participants were employed at the time of the survey. Although two- 
thirds of JC participants enrolled in post-secondary education during their time at JC, only 10 percent 
were enrolled one year after YC completion. Post-program involvement with the justice system was 
relatively limited -- sixteen percent of JC participants were arrested, and 7 percent were convicted of a 
new charge in the time between YC completion and data collection. 


e Court-involved JC participants had similar rates of involvement in a productive activity as non- 
court-involved participants, but had higher rates of criminal justice involvement following the 
program. There was no measurable difference in the post-program employment rates or job 
characteristics of court-involved participants and non-court-involved participants. Although there was no 
measurable difference in postsecondary education during the JC period, by one year following YC 
completion, court-involved participants were less than half as likely to be enrolled than non-court- 
involved participants. Twenty five percent of court-involved participants were arrested in the period 
between YC and data collection, relative to only 11 percent of non-court-involved participants. 


e Relative to participants in YC only, JC participants had lower rates of criminal justice 
involvement and higher rates of enrollment in postsecondary education. Sixty-seven percent of 
JC participants were enrolled in school within six months of YC exit, compared to 8 percent of YC-only 
participants. This difference was not sustained following the JC program period. These findings should 
not be interpreted as causal estimates of the impact of JC as these two groups had different levels of 
pre-program justice-involvement and likely had different interest in pursuing postsecondary education. 


A. Job ChalleNGe participant outcomes 


We begin this chapter by describing the education, employment, and criminal justice post- 
program outcomes of JC participants. These three domains reflect the primary focuses of the JC 
program. Although these outcomes do not present causal evidence of the impact of JC, they are 
informative in illustrating the post-program experiences of participants. In each section, we also 
describe how the outcomes for court-involved participants compare to the outcomes for non- 
court-involved participants. 
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1. Post-secondary education 


Partnerships with community colleges were strategically designed not only to provide education 
services during the JC residential phase, but also to increase participants’ access to and 
familiarity with post-secondary educational institutions. Academic counselors and JC staff also 
supported youth by helping them identify and apply for additional opportunities for education 
and training beyond JC. To assess the post-secondary outcomes of JC youth, we analyzed NSC 
data on YC and JC youths’ enrollment in colleges and universities, including the name and type 
of institution (two-year or four-year), and the degree type (associate, bachelor’s etc.).!? 


Two-thirds of JC participants were enrolled in post- 67 percent of JC participants 
secondary education within six months of completing enrolled in college within 6 months 
YC#4 (Exhibit VIL.1). Given that most participants of completing YC, but only 10 
went straight from YC to JC, this finding suggests that | Percent were enrolled one year 
many youth attended community colleges as part of aren completing Ye: 

the JC program.!> The share of participants enrolled 
within six months of YC completion varied among sites, with 49 percent enrolled in Georgia, 77 
percent in South Carolina, and 84 percent in Michigan, as shown in Appendix Table E.8. There 
are a few factors should be considered in interpreting this variation. First, NSC data may not 
reflect some of the education programs available to JC participants at post-secondary schools. 
Second, sites may have varied in how non-occupational training courses were offered and which 
ones counted for college credit. Finally, partner community colleges may have varied in their 
requirements for enrollment. Given that many JC participants did not have a GED or high school 
degree at entrance, there may have been restrictions on whether they could get college credit. 


Court-involved participants 


Court-involved and non-court-involved participants had similar post-secondary enrollment rates within six 
months of YC completion, but non-court-involved participants were more likely than court-involved 
participants to continue education or receive a certification. Within six months of YC completion, there was 
no measurable difference in the enrollment rates for court-involved participants and non-court-involved 
participants. However, one year following YC, only 5 percent of court-involved participants were enrolled 
relative to 13 percent of non-court-involved participants. Court-involved youth were also significantly less 
likely to receive a credential any time following YC with 36 percent of court-involved participants receiving a 
credential compared to 50 percent of non-court-involved participants. 


Despite the high enrollment rates through JC, only a small share of JC youth continued to obtain 
education following JC. Across sites, only 10 percent of participants were enrolled in post- 


'3 The NSC collects information on student enrollment from more than 99 percent of U.S. colleges and universities. 
For information on the NSC data, see Appendix B. 


'4 VC is used as a reference period for JC outcomes to facilitate comparison to YC participants. 


2 Among participants included in this analysis, consenting JC cohorts 4-6 participants who also participated in YC 
cohorts 4-6, 92 percent of JC participants went straight from YC to JC. 
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secondary education at the one-year mark following YC completion, and only 8 percent were 
enrolled at the two-year mark following the completion of YC. Site-level differences persisted 


Exhibit VI.1. Post-secondary educational outcomes for JC participants (reported in 
percentages) 


Court- Co) exe) U1 ate 
Total Tan yZednic=xe| involved p-value? 


Enrollment in post-secondary education 
Any post-secondary education 


Within six months of YC 67 66 68 

One year following YC 10 5 13 al 

Two years following YC» 8 4 10 
Enrolled in two-year college 

Within six months of YC 67 66 68 

One year following YC 9 5 12 - 

Two years following YC 5 1 8 fe 
Enrolled in four-year college 

Within six months of YC 0 0 0 

One year following YC 1 0 1 

Two years following YC 2 3 2 
Enrolled in private institution 

Within six months of YC 0 0 0 

One year following YC 0 0 0 

Two years following YC 2 3 1 
Within six months of YC 22 14 27 * 
Anytime following YC 45 36 50 
Sample size 304 103 194 


Source: NSC sample weighted data. Data includes educational outcomes through October 2019. 


Notes: Analysis sample includes JC participants in Cohorts 4-6 who attended YC Cohorts 4—6. Statistics on 
enrollment in two-year and four-year colleges include both public and private institutions. Statistics on 
enrollment in private institutions include both two-year and four-year private institutions. Enrollment 
statistics within six months are calculated as whether there was any enrollment for a participant in the 
six months following YC completion. Enrollment statistics at one (two) years following YC are 
calculated as whether the participant was enrolled in school in the semester in which the one (two) 
year point in time follow YC completion occurred. Youth were categorized as having court 
involvement if at baseline they reported ever being arrested, found guilty of a status offense, 
convicted of a crime, or spent time in a juvenile or adult detention facility, or if they were on probation 
or parole at the time of entering YC. Analyses on the full sample include court-involved youth, non- 
court-involved youth, and youth with missing information on court involvement. Seven sample 
members were missing information on court involvement. 


@ Statistical significance is estimated using chi-squared difference tests to compare differences between court- 
involved youth and non-court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 


> Outcomes calculated two years following YC do not include YC Cohort 6. 


¢ Certification data were not available for Michigan participants. Sample is limited to Georgia and South 
Carolina. 
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two years following YC, with 11 percent of Michigan JC participants enrolled relative to only 7 
and 5 percent in South Carolina and Georgia, respectively (Appendix Table E.8). Nearly all of 
the JC participants who were enrolled in school one year following the completion of YC were 
enrolled in a public two-year college. 


Following YC, 45 percent of participants in South Carolina and Georgia received a certification 
reported through NSC by a post-secondary educational institution (data were not available for 
Michigan'®). About half of the South Carolina and Georgia participants who received 
certifications did so within six months of finishing YC, and nearly all of the certifications were 
obtained within one year of YC completion. It is possible that some of the certifications received 
more than six months following YC completion were earned through JC participation, given the 
timing of when youth finish JC and the potential for delays in the granting of certificates. Even 
considering certifications earned at any time, the share of participants who are indicated by NSC 
data to have earned a certification is substantially lower than the self-reported certification rates 
indicated in the follow-up survey, in which 73 percent of participants report earning an 
occupational training credential. One of the many possible explanations for this could be that not 
all occupational certifications were granted by the colleges or considered formal certifications. It 
is also worth noting that even if youth were working toward a two-year or four-year degree, few 
would have obtained it by the end of data collection. 


Consistent with enrollment patterns by site, a higher share of participants in South Carolina 
received a certification than participants in Georgia. Certification data is not available for 
Michigan participants, but because community college enrollment was the highest in that 
program, we would expect the overall certification number to increase if participants from 
Michigan were included. 


Some youth also went on to complete additional education other than enrollment in a college or 
university, and therefore, their enrollment would not be captured by the NSC data. Twenty eight 
percent of youth reported being enrolled in any courses at the time of the follow-up survey, 
which was collected between 16 and 23 months after youth started JC (Exhibit VI.2). This 
percentage is more than three times the number of JC participants observed in the NSC data as 
enrolled in a post-secondary educational institution two years after YC. Examples of education 
captured by the survey in settings other than post-secondary institutions included GED courses, 
additional professional training, and work-based training. 


'© There were no degrees indicated in the NSC data provided for Michigan participants. This finding is inferred to 
be because the Michigan JC site community college partner did not submit degrees earned through JC to NSC. 
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Exhibit VI.2. Enrollment in any classes following JC (reported in percentages) 


Court- Not court- 
Total involved involved p-value? 
Enrollment in any courses 28 22 30 
Sample size 150 53 95 


Source: Follow-up survey weighted data. 


Notes: Analysis sample includes JC participants in Cohorts 4-6. Youth were categorized as having 
court involvement if at baseline they reported ever being arrested, found guilty of a status 
offense, convicted of a crime, or spent time in a juvenile or adult detention facility, or if they 
were on probation or parole at the time of entering YC. Analyses on the full sample include 
court-involved youth, non-court-involved youth, and youth with missing information on court 
involvement. Two sample members were missing information on court involvement. 
Participants were asked, “Are you taking any courses or classes for academic or work-related 
reasons?” 


4 Statistical significance is estimated using chi-squared difference tests to compare differences between 
the court-involved youth and the not court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 


Youth who continued in education following JC reported varying motivations. Almost half of the 
youth (46 percent) who reported being enrolled in classes reported personal interest as a key 
motivation (Exhibit V1.3). Youth also reported taking classes because it was a requirement for a 
current job (36 percent), or because it was a requirement for the work they wanted to do (27 
percent). Other common reasons for 
continuing education included a 
recommendation from a friend, family 
member, or mentor (30 percent) or 
thinking that it would help them get 
another job (32 percent). 


Court-involved participants 


Court-involved youth were more likely to report taking a 
course that was not required but that they thought would help 
them get a job; however, given the large number of 
hypotheses tests and small sample sizes, this finding may be 
spurious. Despite differences in enrollment following JC, 
there were no measurable differences in expectations for 
future education between court-involved and not court- 
involved youth. 


At the time of the follow-up survey 
collection, many JC youth reported that 
they anticipated obtaining more 
education in their lifetime (Exhibit 
VI.4). Despite 98 percent of 
participants entering YC as high school 
dropouts, only 35 percent of participants thought that high school or a GED would be their 
highest level of education. Another 14 percent expected that a vocational/technical diploma or 
certification, possibly the one received through the JC program, would be their highest level of 
education. Nearly half (48 percent) expected to attend at least some college, and 12 percent of 
these respondents hoped to get a graduate degree. Many of the youth who expected to attain at 
least some college education were not currently enrolled. 
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Exhibit VI.3. JC participants’ reasons for continuing education (reported in percentages) 


62 
60% 
50 
46 
43 42 
40% 39 na 
36 
32 Biro 
27 30 28 [}cour-invotved 
23 27 Bint court-involved 
21 20 
20% 
16 
11 
tl 
EI 0 
0% —— 
Personally Requirement for Requirement for Friend, Not required; Relates to | Someone at JC 
interesting work | currently work |wantto family member, thought it would classes | took recommended it 
do do or mentor help me get at JC * 


recommended it _ better job ** 


Source: Follow-up survey weighted data, total: N=40, court-involved: N=10, not court-involved: N=29. 


Notes: Analysis sample includes JC participants in Cohorts 4-6 who reported being enrolled in education 
at the time of the follow-up survey. Youth were categorized as having court involvement if at 
baseline they reported ever being arrested, found guilty of a status offense, convicted of a crime, 
or spent time in a juvenile or adult detention facility, or if they were on probation or parole at the 
time of entering YC. Analyses of the full sample include court-involved youth, not court-involved 
youth, and youth with missing information on court involvement. One sample member was 
missing information on court involvement. Participants were asked, “People get more education 
for different reasons. Please check off the reasons that you wanted to get additional education.” 
Statistical significance is estimated using chi-squared difference tests to compare differences 
between the court-involved youth and the non-court-involved youth. * p < 0.1; ** p < 0.05; *** p< 
0.01. 
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Exhibit VI.4. JC participants’ expectations for future education attainment (reported in 
percentages) 


38 
35 34 
30% 
. Birota 
20% 47 [}cour-invotved 
16 16 15 Bint court-involved 
14 13 14 14 
12 12 11 
10% uu 9 
7 
6 
A a 
o, iim 


No high High school Some college, Two-year Four-year More than _—_- Vocational/ 
school degree degree/GED no degree college college college Technical 
training 


Source: Follow-up survey weighted data, total: N=150, court-involved: N=53, not court-involved: N=95. 

Notes: Analysis sample includes JC participants in Cohorts 4-6. Youth were categorized as having court 
involvement if at baseline they reported ever being arrested, found guilty of a status offense, 
convicted of a crime, or spent time in a juvenile or adult detention facility, or if they were on 
probation or parole at the time of entering YC. Analyses on the full sample include court-involved 
youth, not court-involved youth, and youth with missing information on court involvement. Two 
sample members were missing information on court involvement. Participants were asked, “What 
is the highest grade or degree of school you think you will complete in your lifetime?” Statistical 
significance is estimated using chi-squared difference tests to compare differences between 
court-involved youth and the not court-involved youth. There were no statistically significant 
differences in expected education between court-involved youth and non-court-involved youth. 


2. Employment 


JC training programs were designed to prepare youth to work in occupations with high demand 
and stable pay. We estimated employment outcomes using data collected through the follow-up 
survey. At the time they left JC, half of JC youth who 

responded to the survey said that they had employment lined 
up for after the program (Exhibit VI.5). By the time of the Half of JC participants had a 
follow-up survey, an average of 14 months after the end of Jon lined Upats ent and af 


. percent were employed at 
JC, 81 percent of youth were employed. At the time of the follow-up survey collection 


follow-up survey, the average youth earned $379 per week 
and worked 33 hours in a week. 
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Exhibit VI.5. Employment outcomes for JC participants (reported in percentages unless 
otherwise noted) 


Court- Co) exe) UT ate 

Employment outcomes ae) F-1| involved involved p-value 
Employed following program 

At JC exit@ 50 49 49 

At follow-up survey? 81 78 82 

Average weekly earnings (dollars)> 379 394 374 

Average hours per week (hours) 33 33 32 

Sample size 150 53 95 


Source: Follow-up survey weighted data. 


Notes: Analysis sample includes JC participants in Cohorts 4-6. Youth were categorized as having court 
involvement if at baseline they reported ever being arrested, found guilty of a status offense, 
convicted of a crime, or spent time in a juvenile or adult detention facility, or if they were on 
probation or parole at the time of entering YC. Analyses on the full sample include court-involved 
youth, not court-involved youth, and youth with missing information on court involvement. Two 
sample members were missing information on court involvement. 


4 Statistical significance is estimated using chi-squared difference tests to compare differences between 
the court-involved youth and the not court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 


b Average earnings and hours are calculated across all participants. Participants who were not currently 
employed were assigned a value of zero for both earnings and hours. Statistical significance is estimated 
using t-tests to compare differences between court-involved youth and the not court-involved youth. * p < 
0.1; ** p < 0.05; *** p < 0.01. 


Among those that were employed at the time of the survey, the average JC participant worked 41 
hours each week with a weekly earnings of about $470 (Exhibit VI.6). More than two-thirds of 
employed JC participants (68 percent) received at least some benefits from their job, including 
health insurance, paid time off, paid holidays, sick days, and/or retirement benefits. On average, 
JC participants reported having been at their job for nine weeks at the time of the follow-up 
survey. 


Court-involved participants 
Employment outcomes were similar for youth with court involvement and youth with no court-involvement. 
At JC exit and at the time of the follow-up survey collection, there was no measurable difference in the 
employment rates of the two groups. Among JC participants who were employed at the time of follow-up 
survey collection, there were no measurable differences in employment characteristics, including weekly 
earnings, benefits, hours, and job tenure, between court-involved youth and non-court-involved youth. 


At the time of the follow-up survey, employed youth were asked what type of company they 
work for (industry) and what they do at that company (occupation). The three occupations most 
commonly reported by youth included “transportation and material moving occupations,” “food 
preparation and serving related occupations,” and “installation, maintenance, and repair 
occupations” (Exhibit VI.7). Consistent with this finding, the most common industry for JC 
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participants was in “accommodation and food services,” which accounted for 21 percent of 
participant jobs (Exhibit VI.8). Manufacturing and “public administration,” which includes the 
participants who enlisted in the military, were also common industries. '” 


Exhibit VI.6. Employment characteristics among those employed at follow-up 


Court- Not court- 
Employment outcomes Beir. Tanyced hYC=xe | Tanyce) NYC=re | p-value? 
Average weekly earnings (dollars) 470 508 456 
Average hours per week (hours ) 41 43 39 
Job provides fringe benefits (percentage) 68 62 71 
Average job tenure (weeks) 9 9 9 
Sample size 119 38 79 


Source: Follow-up survey weighted data. 

Notes: Analysis sample includes JC participants in Cohorts 4-6 who reported being enrolled in education 
at the time of the follow-up survey. Youth were categorized as having court involvement if at 
baseline they reported ever being arrested, found guilty of a status offense, convicted of a crime, 
or spent time in a juvenile or adult detention facility, or if they were on probation or parole at the 
time of entering YC. Analyses on the full sample include court-involved youth, not court-involved 
youth, and youth with missing information on court involvement. Two sample members were 
missing information on court involvement. Participants were asked, “People get more education 
for different reasons. Please check off the reasons that you wanted to get additional education.” 


4 Statistical significance is estimated using t-tests to compare differences between court-involved youth 
and the not court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 


'7 Due to the small number of participants reporting employment in each industry and occupation, we do not report 
industry and occupation by court involvement. 
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Exhibit VI.7. Occupations of those employed at follow-up (reported in percentages) 


Jobs Percentage 


Transportation and material moving occupations 


Installation, maintenance, and repair occupations 


Office and administrative support occupations 


Military specific occupations 


Healthcare support occupations 4 


Farming, fishing, and forestry occupations 3 


Building and grounds cleaning and maintenance occupations 2 


Education, training, and library occupations 2 


Computer and mathematical occupations 


Source: Follow-up survey weighted data. 

Notes: Analysis sample includes JC participants in Cohorts 4-6 who reported being employed at the 
time of the follow-up survey. Because some participants have more than one job, the sum of the 
share of participants in each occupation may not sum to 100. Occupations were calculated 
according to the 2010 Standard Occupational Classifications produced by the Bureau of Labor 
Statistics. 
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Exhibit VI.8. Industries of those employed at follow-up (reported in percentages) 


Industry Percentage 


Accommodation and food services 21 
Manufacturing 18 
Public administration 15 
Retail trade 11 
Transportation and warehousing 11 
Construction 8 
Health care and social assistance 5 
Other services (except public administration) 5 
Arts, entertainment, and recreation 3 
Administrative and support and waste management and remediation services 2 
Information 2 
Wholesale trade 2 
Agriculture, forestry, fishing and hunting 1 
Professional, scientific, and technical services 1 
Sample size 120 


Source: Follow-up survey weighted data. 


Notes: Analysis sample includes JC participants in Cohorts 4-6 who reported being employed at the 
time of the follow-up survey. Because some participants have more than one job, the sum of the 
share of participants in each industry may not sum to 100. Industries were calculated according to 
the 2012 NAICS industry classification codes. 


Nearly two-thirds (66 percent) of youth employed at the time 

of the survey said that JC helped to prepare them for their 

job. Despite finding the program helpful in preparing them Court-involved participants 
for jobs, only 9 percent of participants who had a job lined 55858 SSS 
up at the end of the program said that JC staff helped them 
find that first job (Exhibit VI.9). The remaining participants 
either went back to an old job (20 percent) or found jobs on 
their own (70 percent). These analyses, however, do not 
represent the experiences of JC participants who were not 
working at the time of the follow-up survey and may have 
had different perceptions of the JC program. 


Court-involved participants 
reported using similar methods to 
find their positions as youth without 
court-involvement. 
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Exhibit VI.9. Methods employed participants used to find their first job out of JC 
(reported in percentages) 


74 
70 69 
60% 
T 

40% Wrote 
[fcour-invotvea 
Bin court-involved 

26 
20 
20% 18 
[ ii Ey 
| 
aa EI 


Returned to Found job JC staff helped 
previously held independently to find job 
job 
Source: Follow-up survey weighted data, total: N=75, court-involved: N=25, not court-involved: N=48. 


Notes: Analysis sample includes JC participants in Cohorts 4-6 who reported having had a job at the 
end of JC. Youth were categorized as having court involvement if at baseline they reported ever 
being arrested, found guilty of a status offense, convicted of a crime, or spent time in a juvenile or 
adult detention facility, or if they were on probation or parole at the time of entering YC. Analyses 
on the full sample include court-involved youth, not court-involved youth, and youth with missing 
information on court involvement. Two sample members were missing information on court 
involvement. Statistical significance is estimated using chi-squared difference tests to compare 
differences between the court-involved youth and the not court-involved youth. The difference in 
how youth found jobs between court-involved and not court-involved youth is not statistically 
significant at the 0.1 level. 


3. Military 


Because YC and JC were quasi-military style programs run by the National Guard, participants 
were exposed to the employment opportunities associated with joining the military. Preparing 
youth to be able to enlist was also a goal of the YC and JC programs. Although Some YC 
participants who wanted to join the military may have been able to do so immediately after the 
completion of YC, others may not have been accepted to the military given their age, limited 
education, or court involvement. 
For example, the U.S. Army 
requires all recruits to have at least 
a GED; however, the Army enlists 
only a small number of recruits at 
this minimal level of education 
(Powers 2019). College credits 
could help participants qualify for the military. Some participants may have chosen the JC 
program to obtain additional training and experience prior to joining the military. Overall, 13 


Court-involved participants 


There was no measurable difference in military enlistment 
between court-involved youth and non-court-involved participants. 
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percent of JC participants reported being enlisted at the time of the follow-up survey, and 
therefore employed by the military (Exhibit VI.10). 


Exhibit VI.10. Military enlistment (reported in percentages) 


15% 


10% 


5% 


0% 


Total Court-involved Not court-involved 


Source: Follow-up survey weighted data, total: N=150, court-involved: N=53, not court-involved: N=95. 


Notes: Analysis sample includes JC participants in Cohorts 4-6. Youth were categorized as having court 
involvement if at baseline they reported ever being arrested, found guilty of a status offense, 
convicted of a crime, or spent time in a juvenile or adult detention facility, or if they were on 
probation or parole at the time of entering YC. Analyses on the full sample include court-involved 
youth, not court-involved youth, and youth with missing information on court involvement. Two 
sample members were missing information on court involvement. The difference in military 
enlistment between court-involved and not court-involved youth is not statistically significant at 
the 0.1 level. 


4. Any productive outcome 


An overarching goal of JC was to put participants on a path to economic self-sufficiency through 
education, employment, or military enlistment, referred to as productive activities.’ Eighty-six 
percent of youth reported being involved in a productive activity at the time of the follow-up 
survey collection (Exhibit VI.11). This finding represents a large change from the period prior to 
YC, in which only 28 percent of participants were working and none were enrolled in school, as 
discussed in Chapter I. Although this finding does not demonstrate causal evidence of the 
impact of JC, it shows that most JC participants were substantially more productive following 
the YC and JC programs. 


'8 This should be considered descriptive and not a normative statement. There are other socially productive 
activities we do not consider, such as volunteering or caring for family members. 
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Exhibit VI.11. Any productive activity (reported in percentages) 


100% 


75% 


50% 


25% 


0% 


Employment/military and 
education 


Employment 


Bi ccucation 
Wi miitary 


Source: Follow-up survey weighted data, total: N=150, court-involved: N=53, not court-involved: N=95. 
Notes: Analysis sample includes JC participants in Cohorts 4-6. Youth were categorized as having court 


Court-involved participants 


involvement if at baseline they reported ever being arrested, found guilty of a status offense, 
convicted of a crime, or spent time in a juvenile or adult detention facility, or if they were on 
probation or parole at the time of entering YC. Analyses on the full sample include court-involved 
youth, not court-involved youth, and youth with missing information on court involvement. Two 
sample members were missing information on court involvement. The difference in productive 
activity rates between court-involved and not court-involved youth is not statistically significant at 


There was no measurable difference between court-involved youth and non-court-involved youth in the rate 
of productive activity. Further, a lower rate of involvement in a productive activity for court-involved youth 
would not suggest that the JC program was less valuable to them. Because we are unable to learn what 
participants would have been doing if they had not had the option of JC, we cannot tell whether (or the 
extent to which) JC improved outcomes of the court-involved relative to non-court-involved participants. 


5. Case studies 


Looking at average outcomes for JC participants at a single point in time can mask variation over 
time and across participants. To better capture the experiences of youth over time, we conducted 
a monthly survey of youth in JC Cohorts 5 and 6 by text message over the course of six months, 
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starting approximately seven months after the end of JC.!? Results from the survey showed that 
many participants’ working statuses, hourly wages, and enrollment statuses varied substantially 
over the course of six months. Because of the low response rates to the text message survey, we 
only use this data to illustrate examples of employment patterns and not to draw descriptive 
conclusions on the sample.”° 


Exhibit VI.12 shows snapshots of employment and school enrollment for four JC participants. 
These participants were chosen to illustrate the range of experiences. 


e Participant 1 was steadily employed over the course of the text survey. She worked at least 
20 hours each week with her hourly wage in Month 2 actually exceeding her wage in the next 
three months. 


e Participant 2 became employed in Month 3 of the survey and earned steady, slightly 
increasing hourly wages. 


e Participant 3 started the six-month survey employed at $9 per hour. He then experienced a 
gap in employment at Month 3 but became reemployed at a higher wage of $12 per hour in 
Month 4. In Month 6, Participant 3 also enrolled in school. 


e Participant 4 spent most of the survey period neither employed nor in school, with the 
exception of Month 5, when he worked 20 to 29 hours per week for an hourly wage of $7 per 
hour, the federally mandated minimum wage.”! 


These four example participants illustrate the range of employment and school enrollment 
patterns that JC participants experienced over time. For each of these participants, the findings 
about participant outcomes would be different depending on the month they were surveyed. 
Reporting participant averages helps give an overall picture of participant post-program 
outcomes. However, it is important to interpret these results in the context of the range of 
experiences they represent. 


'9 The text message survey began nine months after the completion of JC for Cohort 5 and five months after the 
completion of JC for Cohort 6. For more details on the collection of the text survey data, see Appendix B. 


sie Only 12 participants responded to all six months of the text message survey. Among these responses, some 
included implausible estimates of weekly hours worked or pay. 


2t il wages in Exhibit VI.12 are rounded to the nearest dollar. The federal minimum wage is $7.25 per hour. 
Participant 4 did not earn under the federally mandated minimum wage. 


67 


Job ChalleNGe Final Report Mathematica 


Exhibit VI.12. Examples of JC participant earnings and school enrollment 


ela dale) 


Weekly hours Hourly wage Tale) | (Ye Mi lale-xed Teco) 


Participant 1: Age 17, Female 


Month 1 20-49 $10 yes 
Month 2 20-49 $12 yes 
Month 3 40-49 $11 yes 
Month 4 30-49 $11 yes 
Month 5 30-49 $11 yes 
Month 6 20-29 $12 Yes 
Month 1 0 - no 
Month 2 0 - no 
Month 3 40-49 $11 no 
Month 4 40-49 $12 no 
Month 5 20-29 $12 no 
Month 6 40-49 $12 no 
Month 1 40-49 $9 no 
Month 2 1-9 $9 no 
Month 3 0 - no 
Month 4 10-19 $12 no 
Month 5 40-49 $12 no 
Month 6 40-49 $12 yes 
Month 1 0 - no 
Month 2 0 - no 
Month 3 0 - no 
Month 4 0 - no 
Month 5 20-29 $7 no 
Month 6 0 - no 


Source: Text message survey. 


Note: Hourly wages were rounded to the nearest dollar. No participants in this example reported 
earning less than $7.25 per hour, which was the federally mandated minimum wage when the 
text survey was fielded. Wages in months in which there were zero earnings are shown as “-.” 


Job ChalleNGe Final Report Mathematica 


6. Youth perceptions of what they would have done in the absence of Job ChalleNGe 


Youth responding to the survey provided their perspective on what they would have done 
following YC in the absence of JC (Exhibit VI.13). The highest proportion (35 percent overall) 
indicated they would have looked for a job. Only 14 percent thought they would have joined a 
different program or taken job training courses, and another 13 percent believed they would have 
gone straight to college following YC. Compared with the 67 percent of participants who 
enrolled through JC, this evidence suggests that most youth who received college credit through 
JC would not have obtained college credit otherwise. Finally, 21 percent of youth say they would 
have joined the military in the absence of JC. 


Exhibit VI.13. Self-reports of what participants would have been doing in the absence of 
JC (reported in percentages) 


45 
40% 
35 
30% 2B 
25 Birota 
21 Wi court.involved 
20% Bint court-involved 
15 
10% s—¢ me ; 
6 of 6 7 
5 
Bal gal Boe all ai 
- on BD E | 
Looked for Another Another Community Four-year Military Other 
job * program vocational college college 
training 


Source: Follow-up survey weighted data, total: N=150, court-involved: N=53, not court-involved: N=95. 


Notes: Analysis sample includes JC participants in Cohorts 4-6. Youth were categorized as having court 
involvement if at baseline they reported ever being arrested, found guilty of a status offense, 
convicted of a crime, or spent time in a juvenile or adult detention facility, or if they were on 
probation or parole at the time of entering YC. Analyses on the full sample include court-involved 
youth, not court-involved youth, and youth with missing information on court involvement. Two 
sample members were missing information on court involvement. Statistical significance is 
estimated using chi-squared difference tests to compare differences between court-involved 
youth and the not court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 
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7. Criminal justice outcomes 


Youth who did not complete high school are at a higher risk of 
involvement with the criminal justice system (Sweeten et al. Sivteeh percent of IC 


2009). This risk is even larger for the youth who entered the 


participants were 


program with court involvement, and therefore had a higher arrested following YC 
likelihood of future court involvement. JC programs sought to completion and 7 percent 
prevent youth involvement with the criminal justice system by were convicted of a new 


supporting positive youth development during JC and providing charge 
access to education and occupational training to support 


opportunities following JC. 


Based on state criminal justice administrative records, we find that 8 percent of JC participants 
were arrested for a new crime within one year of YC ending, and 5 percent were convicted for a 
new crime (Exhibit VI.14). Through the end of criminal justice administrative data collection, 16 
percent of JC participants were arrested for a new crime, and 7 percent were convicted for a new 
crime. These rates represent follow-up periods ranging from 13 months for South Carolina’s 
Cohort 6 up to 31 months for participants in Georgia’s Cohort 4 (not shown). Most of the new 
convictions received by JC participants following YC were for public order crimes, such as 
driving under the influence or carrying a concealed weapon, or property crimes, such as burglary 
or larceny (Exhibit VI.15). Drug offenses and violent offenses were less common. 


These data provide evidence to suggest that most JC participants avoided contact with the 
criminal justice system following program. This finding is particularly positive for youth with a 
history of court involvement, who were at higher risk for returning to the criminal justice system 
following the program (see text box on court-involved participants). Youth who were convicted 
of crimes following the program were mainly charged with less severe crimes. Only 1 percent of 
participants were convicted of a violent offense. For self-reported details of criminal justice 
involvement of JC participants following the program collected through the follow-up survey, 
see Appendix F. These results are largely consistent with results from the administrative data.” 


Exhibit VI.14. Criminal justice outcomes for JC participants (reported in percentages) 


Court- I Ko) mrexel 0) ge 


Criminal justice outcomes involved involved p-value? 


Arrested for a new offense 
One year following YC 8 14 5 i 
Anytime during follow-up 16 25 11 


Convicted for a new offense 


>? These data are not included in the appendix because of two major limitations. First, youth who were incarcerated 
or otherwise involved with the justice system likely did not have a means to complete the survey. Second, youth 
may not self-report justice involvement, particularly in ongoing engagements. 
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Court- Not court- 
Criminal justice outcomes involved involved p-value? 
One year following YC 5 9 2 oe 
Anytime during follow-up 7 13 4 id 
Sample size 304 103 194 


Source: Criminal justice weighted administrative data. 

Note: Analysis sample includes JC participants in Cohorts 4-6 who attended YC Cohorts 4-6. YC 
program end dates were considered as May 31, 2017, for Cohort 4, December 31, 2018, for 
Cohort 5, and May 31, 2018, for Cohort 6. Criminal justice data were collected through June 2019 
for South Carolina, November 2019 for Michigan, and December 2019 for Georgia. Youth were 
categorized as having court involvement if at baseline they reported ever being arrested, found 
guilty of a status offense, convicted of a crime, or spent time in a juvenile or adult detention 
facility, or if they were on probation or parole at the time of entering YC. Analyses on the full 
sample include court-involved youth, not court-involved youth, and youth with missing information 
on court involvement. Seven sample members were missing information on court involvement. 


4 Statistical significance is estimated using chi-squared difference tests to compare differences between 
the court-involved youth and the not court-involved youth. * p < 0.1; ** p < 0.05; *** p < 0.01. 


Exhibit VI.15. Conviction outcomes by type of charge (reported in percentages) 


6% : 
5 
4% 4 Biota 
3 Wi courtinvoived 
3 3 | court-involved 
2% 2 
1 1 
1 
‘ En . = im. 
Public order Property Drug Violent Unknown 
conviction conviction conviction conviction category 
conviction 


Source: Criminal justice weighted administrative data. 

Note: Analysis sample includes JC participants in Cohorts 4-6 who attended YC Cohorts 4-6. YC 
program end dates were considered as May 31, 2017, for Cohort 4, December 31, 2018, for 
Cohort 5, and May 31, 2018, for Cohort 6. Criminal justice data were collected through June 2019 
for South Carolina, November 2019 for Michigan, and December 2019 for Georgia. Youth were 
categorized as having court involvement if at baseline they reported ever being arrested, found 
guilty of a status offense, convicted of a crime, or spent time in a juvenile or adult detention 
facility, or if they were on probation or parole at the time of entering YC. Analyses on the full 
sample include court-involved youth, not court-involved youth, and youth with missing information 
on court involvement. Seven sample members were missing information on court involvement. 
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Court-involved participants 


Participants with court-involvement prior to enrollment had higher rates of criminal justice involvement 
following YC completion than non-court-involved participants. Court-involved participants were more than 
twice as likely to be arrested for a new crime (25 percent) and more than three times likely to be convicted 
for a new offense (13 percent) during the follow-up period. Although court-involved youth had higher rates 
of criminal justice involvement, these estimates are lower than most estimates of recidivism rates for youth 
with a history of court involvement. For example, one study finds that one-third of court-involved youth were 
incarcerated by age 25 (Aizer and Doyle 2005). 


B. Comparison to Youth ChalleNGe participant outcomes 


Previous research from the random assignment ChalleNGe Evaluation documented that YC is 
effective at improving outcomes for participants (Millenky et al. 2011). Therefore, one key 
question is what the incremental value is to participants of also attending JC. We are unable to 
assess the impact of participation in JC given the design used for this study and therefore cannot 
draw any causal links between observed outcomes and the effectiveness of JC. However, to 
provide descriptive context, we compare the educational and criminal justice outcomes of JC 
youth with those who participated only in YC. Because employment outcomes are from the 
follow-up survey, which was collected only for JC participants, we are unable present this 
comparison for employment statistics.7* These results should be interpreted with caution given 
that (1) as described in Chapter II, there are differences between the characteristics of YC and 
JC participants, including levels of court involvement, and (2) we cannot account for 
unobservable characteristics such as motivation and social skills or qualitative program selection 
criteria such as staff assessment of youth’s likelihood of success. 


1. Comparison of Youth ChalleNGe and Job ChalleNGe youth under the Job ChalleNGe 
grants 


JC participants were more likely to receive post-secondary education than those who participated 
only in YC (Exhibit VI.16). Only 8 percent of YC-only participants were enrolled in post- 
secondary education within six months of completing YC, relative to 67 percent of JC 
participants. JC participants were also substantially more likely than YC only participants to 
receive a certification—45 percent relative to only 2 percent. Given that most participants go 
straight from YC to JC, both of these differences appear to primarily reflect differences in 
education gained during the JC program. Consistent with this finding, the differences in 
enrollment are only present for public two-year colleges and did not persist over time, with a 
difference of only two percentage points in enrollment between YC and JC participants at one 
year following YC. Despite the fact that there was no difference in educational attainment 


3 We had originally planned to compare employment outcomes using follow-up data collected by programs in the 
YC program administrative data. Unfortunately, these data were incomplete, and there was evidence that 
employment outcomes were biased towards including data on employed participants. 
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Exhibit VI.16. Education outcomes for JC and YC only participants (reported in 
percentages) 


Total p-value? 


Enrollment in post-secondary education 
Any post-secondary education 


Within six months of YC 26 8 67 peed 
One year following YC 8 8 10 

Two years following YC® 10 11 8 

Enrolled in two-year college 

Within six months of YC 25 7 67 or 
One year following YC 7 6 9 

Two years following YC 7 8 &) 

Enrolled in four-year college 

Within six months of YC 1 1 0 on 
One year following YC 1 2 1 

Two years following YC® 3 3 2 

Enrolled in private institution 

Within six months of YC 1 1 0 

One year following YC 0 0 0 

Two years following YC® 1 1 2 

Within six months of YC 6 0 22 oe 
Anytime following YC 14 2 45 sit 
Sample size 984 680 304 


Source: NSC weighted data. 


Notes: Analysis sample includes participants in YC Cohorts 4-6. Statistics on enrollment in two-year and 
four-year colleges include both public and private institutions. Statistics on enrollment in private 
institutions include both two-year and four-year private institutions. Enrollment statistics at one 
(two) years following YC are estimated as whether the participant was enrolled in school in the 
semester in which the one (two) year point in time follow YC completion occurred. 


Statistical significance is estimated using chi-squared difference tests to compare differences between 
YC only youth and JC youth. *p <0.1; ** p< 0.05; *** p < 0.01. 


> Outcomes calculated two years following YC do not include YC Cohort 6. 


© Certification data were not available for Michigan participants. Sample is limited to Georgia and South 
Carolina. 


beyond JC, the credentials earned through JC have the potential to be a valuable signal of 
employability in the labor market (Grossman et al. 2015). 


JC participants were less likely to have criminal justice involvement following YC than 
participants in YC only. Only 16 percent of JC participants were arrested following YC, relative 
to 27 percent of participants in YC only (Exhibit VI.17). Although this difference is large and 
statistically significant, it may represent variation in participant characteristics given that JC 
participants were seven percentage points less likely to have criminal justice involvement prior 
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Exhibit VI.17. Criminal justice outcomes for JC and YC only participants (reported in 
percentages) 


Criminal justice outcomes We) r-1| YC only JC p-value? 
Arrested for a new offense 

One year following YC 12 14 8 sek 
Anytime following YC 24 27 16 ae 


Convicted for a new offense 


One year following YC 5 5 5 
Anytime following YC 8 8 7 
Sample size 984 680 304 


Source: Criminal justice weighted administrative data. 


Note: Analysis sample includes participants in YC Cohorts 4-6. Because participants could be 
convicted of more than one type of crime, the sum of the percentages of participants who were 
convicted of the four crime types (drug offenses, violent offenses, property crimes, and public 
order crimes) could be greater than the percentage of participants convicted of a new offense. 
Rates represent the share of participants with a criminal justice event anytime following the YC 
program. YC program end dates were considered as May 31, 2017, for Cohort 4, December 31, 
2018, for Cohort 5, and May 31, 2018, for Cohort 6. Criminal justice data were collected through 
June 2019 for South Carolina, November 2019 for Michigan, and December 2019 for Georgia. 


Statistical significance is estimated using chi-squared difference tests to compare differences between 
YC only youth and JC youth. *p < 0.1; ** p< 0.05; *** p < 0.01. 


to YC. There was no measurable difference in conviction rates of JC and YC-only participants. It 
is worth noting that the conviction rates for both groups were low, with less than 10 percent 
being convicted of a new crime by the end of data collection. 


To assess the degree to which differences in observable characteristics between JC and YC-only 
participants explain the differences in outcomes, we ran a regression analysis controlling for key 
observable characteristics of participants. We found that the key takeaways did not change after 
controlling for observable characteristics. Although this does not account for the full set of 
differences between JC and YC-only youth, it provides evidence on the extent to which between 
group differences can be explained by observable characteristics collected in this study. The 
results of this analysis are presented in Appendix D. 


2. Comparison to prior literature 


Prior research on the outcomes of YC participants can also provide benchmarks for expected 
outcomes of JC participants. We compared the outcomes for JC participants to the findings of the 
ChalleNGe Evaluation (Millenky et al. 2010). The ChalleNGe Evaluation examined outcomes of 
participants at 10 sites and 18 unique YC cohorts in 2005 and 2006. The sites were selected to be 
as representative of YC sites overall as much as possible. 
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There are important qualifications to consider in interpreting a comparison of JC outcomes to 
previous YC impact study. First, there are meaningful differences in participant characteristics, 
economic conditions, and program specifics between the JC sites and the sites included in prior 
research (Millenky et al. 2010). Second, the YC outcomes found in the prior study are an average 
of the outcomes of participants who would and would not have participated in JC if it had been 
available. If high-achieving YC participants are more likely to join JC and the JC program 
increased participants’ outcomes, we could not disentangle the effects of each of these factors 
when interpreting differences in average outcomes for JC participants in this study and YC 
participants in prior studies. Finally, the prior study of YC was conducted in a different time 
period with different labor market conditions and job-market opportunities for youth. Despite 
these limitations, prior research can provide an informative benchmark for JC participant 
outcomes as YC participants have many similarities to JC participants. 


Exhibit VI.18 compares the demographic characteristics of youth entering YC in this study with 
those of youth who participated in the prior study of YC (Millenky et al. 2010). We do see some 
differences in the samples. Twenty percent of youth in the ChalleNGe Evaluation were Hispanic 
compared to eight percent of the YC-only youth in this study. They also had completed less 
education prior to YC than YC-only participants at JC grantee sites. Despite the JC grant focus 
on recruiting participants with court involvement, slightly more YC participants at the 
ChalleNGe Evaluation sites reported having ever been arrested or convicted of a crime prior to 
YC entrance. 


Exhibit VI.18. Comparison of JC youth characteristics with prior research (reported in 
percentages unless otherwise specified) 


ChalleNGe 
Evaluation sample of 
F-Torer-1 0} (:10 my 03 YC-only 
participants participants JC participants 

Male 88 82 77 
Average age 17 16 17 
Race and ethnicity 

Hispanic 20 8 8 

Non-Hispanic, black 31 53 43 

Non-Hispanic, white 43 29 39 

Non-Hispanic, other race 0 9 10 
Highest grade completed 

8th grade or below 12 8 6 

9th grade 32 27 23 

10th grade 36 39 42 

11th grade 19 21 24 

12th grade 1 4 5 
Ever suspended from school 82 82 74 
Ever arrested 34 30 28 
Ever convicted 19 20 15 
Sample size 736 680 304 

Source: ChalleNGe Evaluation (Millenky et al. 2010), Table A.3, background information form weighted 
data. 
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For the outcomes comparison, we used outcomes from the 21-month follow-up survey conducted 
by Millenky et al., which roughly lined up with the timing of our follow-up survey. Overall, JC 
youth were significantly more likely (14 percentage points) to be involved in any productive 
activity. JC participants were three times as likely to have ever enrolled in college and more 
likely than participants in the Millenky et al. study to receive any college credit, likely because 
JC participants enrolled in college through the JC program (Exhibit VI.19). JC participants were 
also 26 percentage points more likely to be employed in the follow-up period. 


Exhibit VI.19. Comparison of JC participant outcomes to prior research (reported in 
percentages unless otherwise specified) 


ChalleNGe Evaluation JC participants 
¥-Tan] eo) (= Mey mr-(e1ex-) el (=1e| 
YC participants? YC-only 
Any productive activity*4 72 - 86*** 
Post-secondary education 
ae oo any 25 17 75*** 
Currently enrolled in college 12 11 8 
Currently enrolled in any courses¢ 34 - 28 
Employment 
Currently working 55 - 81*** 
Average weekly earnings (dollars)°9 261 = 379*** 
ice yee Gore te “ey 
Military 
Currently enlisted® 11 - 13 
Crime and delinquent activity 
Arrested" 26 27 16** 
Convicted" 9 8 7 


a Source: ChalleNGe Evaluation (Millenky et al. 2010), N = 736. 


> We conducted statistical tests comparing the JC participant outcomes to the Millenky et al. (2010) 
participant outcomes. * p < 0.1; ** p < 0.05; *** p < 0.01. 


¢ JC source: Follow-up survey. N = 150. Analysis sample includes JC participants in Cohorts 4-6. 
4 Any productive activity is defined as employment, education, or military enlistment. 


¢ YC-only and JC source: NSC weighted data. YC only: N = 680, JC: N = 304. Analysis sample includes 
participants in YC Cohorts 4-6. 


f Currently enrolled in classes is defined for JC participants as enrollment at the point in time two years 
following the end of YC to most closely match the Millenky et al. (2010) survey timing. 


9 Shown in 2019 real dollars based on CPI-U. 


" Source: Criminal justice administrative weighted data. YC only: N = 680, JC: N = 304. Analysis sample 
includes participants in YC Cohorts 4-6. 


The differences in college enrollment between the two studies are large and therefore unlikely to 
be explained by differences in youth characteristics. This finding is supported by the fact that 
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most JC participants reported that they did not think they would have enrolled in college in the 
absence of JC. Given differences in youth characteristics and motivation as well as economic 
circumstances, it is unclear whether the high employment rates of JC participants compared with 
the Millenky et al. sample imply that the program was effective. However, the results suggest 
that this area is promising for future research to assess the impacts of JC 
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Vil. LESSONS FROM THE JOB CHALLENGE EVALUATION 


The U.S. Department of Labor (DOL) piloted JC as an innovative strategy that builds upon the 
successes of the National Guard Youth ChalleNGe program. As the earlier ChalleNGe 
Evaluation described, YC participants were more likely to have earned a GED or completed high 
school, earned college credits, and be employed, but youth reported that it was hard to maintain 
momentum and carve out a path forward as they returned to their communities. To continue 
supporting YC graduates, the JC program provides them with the opportunity to further their 
education and vocational training in preparation for a career. 


In April 2015, DOL launched the JC program with three grants to YC programs in South 
Carolina, Michigan, and Georgia. These grants had two goals: to improve youth education and 
employment outcomes by (1) increasing access to YC for court-involved youth and (2) 
developing the new occupationally focused JC program, which built on the YC environment 
while offering participants more freedom and new opportunities. As voluntary and free programs 
that offered a combined 42 weeks of residential programming, YC and JC offered a unique 
opportunity for young people to not only “get back on track” but build skills for a successful 
career. 


All three grantees launched new residential programs and developed partnerships with 
community colleges to offer participants occupational training and access to a college 
experience. The leadership for each program identified nearby lodging, hired program staff, and 
coordinated closely with their companion YC program(s) to select among YC graduates. Overall, 
the three JC programs recruited and enrolled 905 youth between January 2016 and December 
2018, including 333 in Georgia, 301 in Michigan, and 271 in South Carolina. This total exceeded 
DOL’s overall target of 900 participants. 


In this chapter, we summarize our key findings on program implementation, the outcomes of JC 
participants, and considerations for the future. 


A. Findings on program implementation 


DOL had two distinct goals for the JC grants: (1) to provide more court-involved youth with 
access to an evidence-based youth program and (2) to implement a new occupationally intensive 
program for YC graduates to further support youth development and prepare youth for the labor 
market. For grantees, the primary focus was the substantial undertaking of establishing the new 
JC program. Grantees continued to serve court-involved youth and placed a higher priority on 
documenting court involvement, but they did not substantially alter YC recruitment practices to 
reach new groups of participants or adjust service delivery for court-involved youth. 


1. Increasing access to YC for court-involved youth 


Programs reported difficulty reconciling the focus of the DOL grant with the existing 
National Guard criteria for Youth ChalleNGe. The National Guard eligibility criteria 
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explicitly states that participants may not be currently on parole or probation for other than 
juvenile status offenses, awaiting sentencing, under indictment, or accused or convicted of a 
felony. Further, it states that participants cannot have any pending court dates once 

the program starts. Without a clear definition from DOL and an agreement with the National 
Guard that allowed a shift from the existing criteria, programs defined court involvement as they 
saw fit. 


Programs already had recruitment partners in the justice system and continued to draw on 
them. YC programs reported receiving referrals from the Department of Juvenile Justice and 
family courts. YC programs also collaborated with local judges who made referrals as part of 
diversion programs. 


YC staff did not perceive a need to shift programmatic practices, because they already 
served young people who fit the definition of “court-involved.” Programs did not find it 
appropriate or necessary to single out court-involved youth during their time in YC or JC. They 
believed they were already serving this population, even if at a slightly lower rate, and that the 
existing programming met their needs. YC program staff also expressed concern about their 
program being labeled as a program targeting court-involved youth. 


2. Implementation of JC program 


Having a large YC partner program facilitated recruitment. JC programs with smaller YC 
feeder programs reported greater recruitment challenges. While two JC programs recruited from 
a single YC program in their respective states, Georgia recruited from three YC programs within 
the state, and its YC partner program was substantially larger than the YC programs in South 
Carolina and Michigan. This meant that the Georgia JC program had a larger pool of qualified 
applicants hence allowing them to be more selective and to give priority to court-involved youth. 


Although programs prioritized court-involved youth in the JC application process, the 
grantees did not reach the DOL performance target of 50 percent court-involved youth. 
Among study participants, JC youth were actually less likely to be court-involved than YC 
youth. We were not able to determine whether this difference reflected interest in the JC program 
or JC eligibility. 


Programs found that they needed more supervision of youth than initially planned. At the 
start of JC, program staff gave JC participants substantially more freedom than during YC, as 
they expected that YC graduates would have acquired a higher level of maturity. The housing 
mirrored this approach by offering youth at least some level of privacy that was not available in 
YC. However, that approach evolved over time as staff realized that participants did not yet have 
the self-discipline and still needed a more structured environment. Even with this change, JC 
offered a more relaxed environment than YC. 


JC program administrators found that they needed more resources than originally 
anticipated for the residential program. They reported that additional resources were needed 
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to support the cost of food, staffing, and the location of the housing. For housing, proximity to 
the partner community college and to YC was important and played into both costs and other 
logistics. Staffing needs, as noted above, were lower than for YC, but higher than programs had 
budgeted for in their DOL grants. However, JC programs on military bases (where many YC 
programs are located) could leverage cost-cutting in other areas. 


JC programs relied on close partnerships with community colleges to deliver vocational 
training and provide career counseling to youth. Program staff reported that, although youth 
liked the opportunity to be a part of the college, there was a benefit to having some JC-student- 
only courses at the community colleges. It provided an opportunity for JC participants to adjust 
to the college environment, and the pace of the course and the approach to discipline could be 
shifted to the needs of the JC program. The disadvantage of this approach was that youth 
reported in focus group discussions that separate sections made them feel less like they were part 
of the college. 


Programs struggled to offer youth work-based learning opportunities. Program staff 
reported that they had to get creative to offer youth exposure to a workplace, because so many 
participants were minors. In the survey of JC participants, job shadowing and field trips were 
approximately twice as common as internships. 


B. Findings on Job ChalleNGe participant outcomes 


The primary goal of JC was to help prepare youth for employment. Grantees offered youth 
access to intensive occupational training courses at partner community colleges, as well as 
additional secondary and postsecondary courses. Using survey data and administrative data on 
postsecondary education and criminal justice involvement, we examined outcomes as many as 
14 months after JC. 


Most JC participants found employment following the program, and many felt that JC 
helped prepare them for these jobs. By the time of follow-up survey collection, approximately 
14 months after JC, more than 80 percent of JC participants were employed. Among those 
employed, 66 percent said they felt JC helped prepare them for their job. 


Court-involved JC participants achieved similar rates of employment as non-court- 
involved participants. Among JC participants who were employed at the time of follow-up 
survey collection, there were no measurable differences in employment characteristics— 
including weekly earnings, benefits, hours, and job tenure—between court-involved youth and 
non-court-involved youth. 


Youth had relatively low levels of involvement with the criminal justice system. Using state 
criminal justice administrative records, we found that 8 percent of JC participants were arrested 
for a new crime within one year of YC ending, and 5 percent were convicted of a new crime. 
Participants who were court-involved before enrollment were more likely to be arrested for a 
new crime (14 percent) and be convicted for a new offense (9 percent) during the same one-year 
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period after YC. Although the rate of criminal justice involvement is substantially lower than 
most estimates of recidivism rates for youth with a history of court involvement (Aizer and 
Doyle 2005), youth who choose to attend YC and JC may be different than typical youth with 
court-involvement. 


There is also evidence to suggest that JC improved education outcomes beyond what youth 
would have achieved without the program. Although this study cannot provide causal 
evidence, JC program participants enrolled in post-secondary education at a higher rate than YC 
participants who did not continue to Job ChalleNGe. JC participants were more than eight times 
as likely to enroll in post-secondary education within six months of completing YC (67 percent 
compared with 8 percent). JC participants were also substantially more likely than YC only 
participants to receive a certification—45 percent relative to only 2 percent. Both of these 
differences reflect differences in education gained during JC, as only 10 percent of JC 
participants were enrolled in post-secondary education at the one-year mark following YC 
completion. 


C. Considerations for the future 


Policymakers, practitioners, and researchers alike are broadly focusing on the type of 
programming offered through Job ChalleNGe (Pollack 2017). A high school diploma or high 
school equivalency credential is not enough to put a young person on the road to success and 
financial security in adulthood (Clark and Martorell 2014). 


At this report’s release, the Job ChalleNGe program continues in the three pilot sites, without 
funding from DOL, plus a few additional locations around the country. This study offers lessons 
that can inform current and future programming for Job ChalleNGe. The pilot program was 
unique in that it layered the DOL-funded JC program atop the DOD-funded Youth ChalleNGe 
program. This approach created additional perceived challenges for the programs as they tried to 
follow guidelines, funding requirements, and performance goals from different federal agencies. 
Going forward, JC is a DoD-led initiative, so JC programs will have the increased flexibility of 
working under only DoD guidelines. 


Although the outcomes for JC participants are encouraging, this study does not provide evidence 
on the effectiveness of JC program. Understanding the impact of JC on the employment, 
education, and criminal justice outcomes of youth participants requires an impact study, but 
aspects of the JC program, including small cohorts and enrolling from a fixed population of YC 
graduates, make it difficult to conduct a random assignment study. As the number of JC 
programs increases, there may be additional opportunities to measure the program’s impact 
including evaluation designs that compare the outcomes of JC participants to the outcomes of 
similar youth enrolled in other YC programs that do not have access to JC. 
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 crantee Profile: Georgia Job ChalleNGe 


Georgia Contextual Program AZ2 
Job ChalleNGe factors services A 
Program name: Georgia Job Located on the same Army base Occupational training tracks: Welding, culi- 
ChalleNGe Program as aYC site, which facilitated use nary arts, apartment/hotel management, auto- 
Grahtec want Natousl Gaara of YC staff and other resources motive repair, and certified nursing assistant 
Youth ChalleNGe(YC)/Job Atmosphere of a large military Certificate/credentials available: Diploma/ 
ChalleNGe (JC) Academy base with cadets housed in GED, Occupational Safety and Health 
JC grant emonntrs).060.000 barracks with slightly more Administration (OSHA), cordiopulmonary 
privacy than in YC and with resuscitation (CPR), Microsoft Office, certifi- 
JC location: Fort Stewart, Georgia limited ability to use base cates in each field 
facilities 


Partner YC site: Georgia 
National Guard Youth ChalleNGe 


Work-based learning: Opportunities 
included field-based training such as prac- 


Academy, Fort Stewart, Georgia 


Occupational training partner: 
Savannah Technical College 


YC and JC enrollment 


Percent 
fo) Morel ela 
la\e)ii-e Ra al-1s | 


| Percent of all 
youth that 
fete} as} e)(-3 4-10! 


Percent 
court 


Number 
of youth 
completed 


enrolled | involved 


Source: Grantee performance reports that JC grantees submit- 
ted to DOL between September 2017 and December 2018. The 
completion numbers contained in the performance reports 
cannot be verified using other available data sources. 


Definition of court-involved youth: Anyone involved in 
the court system, excluding foster youth. Only individ- 
uals with juvenile offense histories. Applicants could 
not be on probation unless they received a letter stating 
probation would be suspended during participation. 


tice climbing commercial communications 
towers, working in medical facilities, or 
other opportunities such as hearing from 
guest speakers and going on field trips. 


Non-Academic services; Opportunities 
included work readiness and life skills 
taught through a for-credit college class, a 
career pathways course provided by work- 
force system partners; career and academic 
counseling through YC counseling staff; and 
leadership opportunities, including super- 
vising cadet activities, representing JC in the 
community, and serving on the youth council. 


Demographic characteristics of 
JC participants at enrollment 
6% 


Hispanic] 


y 


| 83% 


55% 
Black 


12% 
Other/ 
unknown 


64% 
Male’ 


27% 
White High school 
graduate/GED 


Source: Grantee performance reports. 
114 percent of Georgia’s JC youth had missing values for gender. 
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W Grantee Profile: Michigan Job ChalleNGe 


O Eo 
Michigan Contextual Program [s 
Job ChalleNGe factors services oa! 
Program name: Michigan Job Located a few miles from its Occupational training tracks: Informa- 
ChalleNGe (JC) Program partner YC site, which admin- tion technology; heating, ventilation, and 
Grantantiecna: Might, enUepart? istrators report made using YC air conditioning (HVAC), medical first 
ment of Military and Veterans staff and program resources responder, certified nursing assistant, weld- 
Affairs peat sitaer but presented ing, electrical, pipe fitting, and robotics 
some challenges ‘ 
a Giant enemas esbesIeS g Certificate/eredentials available: Diploma/ 
Located ona National Guard GED, Occupational Safety and Health Adminis- 
JC location: Fort Custer Training training facility with cadets tration (OSHA), cardiopulmonary resuscitation 
Center, Battle Creek, Michigan housed in dorm-like accommoda- (CPR), Microsoft Office, certificates in each field 
Partner YC site: Michigan Youth Sons - the base hotel, which made Work-based learning: Opportunities 
ChalleNGe (YC) Academy, Battle 38 Soi slice i renee ere included field-based training such as prac- 
Creek, Michigan (VA Campus) pubeiee Sf praises ie a tice climbing commercial communications 
regular military personnel ae We ; Gis 
Occupational training partner: . towers, working in medical facilities, or 
Kellogg Community College other opportunities such as hearing from 
(Regional Manufacturing and guest speakers and going on field trips. 
Training Center) Non-Academic services: Opportunities 
scHIGAM YOU; included work readiness and life skills 
. 7 taught through a for-credit college class; 
a * career and academic counseling through JC 
es Jey program staff, and leadership opportunities, 
. * including supervising cadet activities and 
Sh ptptene representing JC in the community. 


YC and JC enrollment 


yc 


928 33% 


301 43% 


Percent 


63% 


Demographic characteristics of 


79% 


Source; Grantee performance reports that JC grantees submit- 


17% 
Other/ 
unknown 


JC participants at enrollment 


Number | Percent fe) Motel tla Percent of all 
of youth| court involved that | youth that 6% es 
| enrolled | involved | completed forelas|e)(-\i-1e! Hispanic 52% 


White 


ted to DOL between September 2017 and December 2018. The 
completion numbers contained in the performance reports 
cannot be verified using other available data sources. 

NA = not available 


ie 41% 


High school 
graduate/GED 


Definition of court-involved youth: Anyone who has 
been formally adjudicated, arrested, or charged. Also 
includes anyone in the foster care system or family 
court system, No open or pending cases or youth on 


Source: Grantee performance reports. 
probation are allowed. 


16 percent of Michigan's JC youth had missing values for gender 
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Grantee Profile: South Carolina 


Job ChalleNGe 


South Carolina Contextual Program jag 
Job ChalleNGe (JC) factors services A 
Program name: South Carolina Located about 70 miles from Occupational training tracks: Tower tech (ell 
Youth ChalleNGe (YC) Academy its partner YC site, which phone tower climbing and repair), patient care 
POST ChalleNGe Program administrators reported made assistance, computer networking and installa- 
Grautes namer South Garcling it challenging to use YC staff tion, welding, and production/operations 
Military Department end resources Certificate/credentials available: Diploma/ 
Rural environment at a university GED, Occupational Safety and Health Adminis- 


JC grant amount: $3,994,475 
JC location: Aiken, South Carolina 


retreat center, with access to 


Partner YC site: South Carolina 
Youth ChalleNGe Academy, 
Eastover, South Carolina (McCrady 
Training Center, Fort Jackson) 


with bunk beds 


Occupational training partner: 
Aiken Technical College 


youth Chay, 
‘a, 
uo 


South o 
oy 
re) 
Awape™ 


4 ae 


YC and JC enrollment 


Percent 
fo) Move] Fla 
Ta\elhi-e Raal-1s 
(ote) ante) (-) 4-10) 


Percent of all 
youth that 
foto} ante} i-i<-te! 


Percent 
court 
involved 


Number 
of youth 
Tal ge) i(-re} 


Source: Grantee performance reports that JC grantees submit- 
ted to DOL between September 2017 and December 2018. The 
completion numbers contained in the performance reports 
cannot be verified using other available data sources. 

NA = not available 


Definition of court-involved youth: Anyone who has 
been involved with the juvenile system, adult system, or 
social services (homeless or foster youth). Cannot have 
a violent offense. Open to individuals with pending 
juvenile charges. 


ropes courses and outdoor activ- 
ities; students housed in cabins 


tration (OSHA), cardiopulmonary resuscitation 
(CPR), Microsoft Office, certificates in each field 


Work-based learning: Opportunities 
included field-based training such as prac- 
tice climbing commercial communications 
towers, working in medical facilities, or 
other opportunities such as hearing from 
guest speakers and going on field trips. 


Non-Academic services: Opportunities 
included work readiness and life skills taught 
in a for-credit college class and supplemental, 
staff-organized classes; career and academic 
counseling through JC program staff: and 
leadership opportunities, including supervis- 
ing cadet activities. 


Demographic characteristics of 
JC participants at enrollment 


other! 

r 

Salon 62% 
5% f Black 

Hispanic 


i = 32% 
High school 
graduate/GED 


Source: Grantee performance reports. 
1One percent of South Carolina’s JC youth had missing values 
for gender. 
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In this appendix, we describe quantitative data collection and the contents of the four types of 
data sources used in this report. First, we describe the background and contact information data, 
which were collected in the background information form. Next, we describe the administrative 
program data that grantees and DOL provided for the purposes of this study. We then describe 
the administrative outcomes data, including data on criminal justice outcomes obtained from 
state criminal justice agencies and data on educational outcomes from the National Student 
Clearinghouse (NSC). Finally, we describe the survey data collected through the follow-up 
survey of participants and the text message survey. 


BACKGROUND INFORMATION FORM 


Background information forms (BIFs) were collected from all participants participating in the 
study. In addition to the BIF, all participants returned a completed youth consent form and an 
additional parent consent form if they were younger than 18 at the time of enrollment in the 
study. The consent forms, for both parents and youth, described the evaluation and ensured that 
the study participants had been fully informed about the implications of participating in the 
study, the data collection elements, and data security. 


For Cohorts 4—6, Mathematica trained a group of field staff in each location on the process for 
collecting BIFs and consent materials. Field staff then visited YC and JC programs during intake, 
visitations, or family days, when it was expected parents would already be on-site to complete 
the parent consent form (where applicable). This method was developed after our collection of 
pilot data in earlier cohorts revealed logistical issues with having YC and JC program staff 
collect consent materials and BIFs. 


Forms were to be distributed as close to the start of the YC program as possible. However, due to 
the challenges program staff faced with collecting BIFs and limited windows for field staff visits, 
BIF collection dates varied relative to the start of each program. In some cases, if the BIF could 
not be collected over the course of YC, it was collected at the beginning of JC. 


The BIFs collected demographic information and information on prior educational attainment, 
prior employment, delinquency, and criminal justice involvement history. We also collected 
multiple forms of contact information, including social media contact information, to help ensure 
successful follow-up data collection in the future. Regardless of when youth completed the form, 
the questions on prior education, employment, and delinquency referenced the time period before 
youth began YC. 


For estimates of the share of youth who completed a BIF, see Appendix C. 
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PROGRAM DATA 


1. Youth ChalleNGe program data 


The YC program data were collected by program staff through the National Guard Cadet 
Tracking system—a robust management information system that all YC programs used. Through 
this, YC programs collected information on participant characteristics, the duration of program 
involvement, program completion, the types of services received, and outcomes for youth at 
varying intervals in the 12 months following the program (including education, employment, and 
military enrollment). Participant characteristics available in the program data include gender, 
date of birth or age, and race. Michigan and South Carolina also provided an indicator for court 
involvement. 


The YC program data presented several key limitations. First, although the data format and 
content were similar across sites, notable differences existed across sites, which limited our 
ability to conduct pooled analyses of all three sites. For example, sites all provided data on 
participant test scores, but the tests on which they reported varied. Michigan reported both the 
Test of Adult Based Education (TABE) and the Armed Service Vocational Aptitude Batters 
(ASVAB) score at entrance, South Carolina reported only the ASVAB score, and Georgia 
reported only the TABE score. Second, Georgia did not provide adequate identifying information 
for matching data on participants from the program data to other study data on participants. 
Finally, South Carolina only provided program data for participants who completed the YC 
program. 


Given these limitations, we were unable to use the program data for much of our analysis. For 
example, we could not use program data on services in analyses, except for high school/GED 
completion. Similarly, data on participant post-program outcomes were not standardized and 
were underpopulated. There was also evidence that the coverage of the program data on 
participant post-program was nonrandom, with participants with a history of employment more 
likely to have program data on employment outcomes. As a result of this bias, small sample sizes 
due to missing data, and the inability to link Georgia participants to the YC data, we did not use 
program data to assess post-program outcomes. 


We were able, however, to use the data to compare the participant characteristics for consenters 
and nonconsenters to assess the risk of nonconsent bias and create consent weights (see 
Appendix X for details). We used the following participant characteristics for this analysis: 
gender, date of birth or age, race, and court involvement (Michigan and South Carolina only). 
We also used participant characteristics to supplement data collected through the BIF to create 
data requests for data on criminal justice outcomes and the NSC data. 


2. Job ChalleNGe program data 


Program data for JC participants was collected by JC staff separately from YC data at each 
program rather than using a standardized collection system. Each site provided the study with JC 
rosters but did not all provide the same information. South Carolina provided cohort-specific 
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rosters with minimal additional information. Georgia provided JC rosters with data on JC 
training program and JC completion. Michigan provided program data in a similar format to the 
YC program data, as tracked by the National Guard Cadet Tracking system. 


We used the JC program data to identify which youth in the study actually participated in JC. 
Although each site gave us this information before the start of each JC cohort, many youth who 
were expected to participate in JC did not end up participating. Additional information from the 
JC program data was not used in study analyses because it was not documented consistently 
across the sites. 


GRANTEE PERFORMANCE REPORTS 


For performance reporting purposes, DOL gave grantees a template to track aggregate measures 
of YC and JC program progress on a monthly basis. The spreadsheet tracked the following 
measures with columns for the current cohort as well as all youth enrolled program-to-date: 

e Enrollment overall 

e Enrollment for court-involved youth 

e Completion for court-involved youth 

e General demographics of the youth 

e Overall YC completion rate 

e Overall JC completion rate (not required but two states tracked it) 

e Post-program outcomes for JC participants on employment, education, or military 

In this study, we used data from the grantee performance reports for two purposes. First, we used 
the data to report overall participant demographics in the grantee profiles. Second, we used the 
performance reports to develop the completion rate statistics. An important caveat is that the 


completion statistics from the performance reports are incomplete and contain inconsistencies 
due to issues described below. 


We intended to use data from the grantee performance reports to describe post-program 
outcomes for participants, but this was not possible because of the following data issues: 


e Inconsistent use of the template. Grantees from different programs filled out the reporting 
spreadsheet in different ways. For example, all reports were missing information for some 
fields, but those fields varied. Additionally, some staff reported that different staff within a 
site did not always complete the template in the same way. 


e Incomplete or missing data. Many of the monthly reports from grantees were missing data. 
For example, in one site, data on the total number of people who completed YC overall were 
missing. In other sites, data on the number of JC participants without court involvement who 
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completed the program were missing. In general, the template provided to grantees did not 
adequately capture YC or JC completion data for non-court-involved youth. 


e Unreliable data within the spreadsheet. Across all grantees, inconsistencies were evident 
in the data across months, or even within a month. For example, in all states, there were 
months for which the sum of the total number of youth who completed the program and the 
total number of youth who dropped out of the program or were removed was a number that 
was higher or lower than the total number of youth enrolled overall. 


Because of these data quality issues, we refrained from using the performance report data for any 
primary analyses. We were not able to use these data to validate the survey or program data or to 
describe completion rates from non-court-involved participants in YC or JC. 


CRIMINAL JUSTICE DATA 


We collected data on criminal arrests and convictions for youth following YC from state criminal 
justice agencies. To obtain these data, we submitted a list of all YC participants in Cohorts 4—6 
who consented to study participation at each JC grantee site to the criminal justice agency in the 
state where the program was located. Each state agency matched YC participants to the database 
of adult criminal justice outcomes using probabilistic matching algorithms based on Social 
Security number, name, date of birth, and sex. Each agency then returned a data file with the 
criminal justice outcomes for each participant. Exhibit B.1 shows the number of months for 
which we collected criminal justice data for each cohort and site. 


Exhibit B.1. Months of coverage of criminal justice administrative data, by site and cohort 


State and criminal justice data end date 


CT-Yo) xe F-] Michigan South Carolina 
YC Program end date 
exe) ate) ai considered DY =Yex=Y 4) ol-1 a4 bo) November 2019 June 2019 
4 May 31, 2017 31 months 30 months 25 months 
5 December 31, 2017 24 months 23 months 18 months 
6 May 31, 2018 19 months 18 months 13 months 


These data present three key limitations: 


1. Limited to criminal justice events in these three states. Because we only submitted data 
requests to the three states with JC programs, criminal justice outcomes do not cover out-of- 
state criminal justice events. Although all of the participants lived in state at YC entrance, 
some participants might have moved out of state after leaving the program. 
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2. Differences in types of data states provided. For example, Michigan did not provide data 
on arrests that had been adjudicated but for which there was no conviction. The data each site 
provided are described in more detail below. 


3. Differences in definition of ‘adult’ across states. Data only cover charges for which the 
criminal justice system considered the participants as legal adults. In Georgia, people are 
tried as adults for any crime if they are 17 and older” or if they are charged with certain 
violent crimes and are 13 and older.” In Michigan, people are tried as adults if they are 17 
and older.*° In South Carolina, for most of our sample period, people were tried as adults for 
any crime if they were 17 and older, but this law changed on July 1, 2019, to raise the age to 
18.77 South Carolina youth older than 14 can be tried as adults for certain felony charges, and 
youth of any age can be tried as adults for murder or criminal sexual conduct. They may also 
be tried as adults if they are charged with a felony or if they have been previously charged 
with a felony.”® 


We estimated outcomes using the following data from each state agency: 


e Georgia. Georgia records were provided by Research Provider Applied Research Services 
from the Georgia Crime Information Center, which serves as the state repository for 
information on Georgia’s criminal history records. Individual records included arrest dates, 
arrest offense categorizations, disposition dates, conviction offense categorizations, final 
dispositions, and sentencing information. Personally identifiable information was stripped 
from these records and replaced with unique, randomly generated ID numbers. Information 
on charges included all Georgia charge codes and descriptions. Data were provided through 
December 2019. 


e Michigan. Michigan records, obtained from the Michigan State Police, were provided across 
six files: demographic information, supplemental demographics, incident, arrest, charge, and 
judicial. Arrest and judicial files were combined using individual criminal tracking numbers 
to link arrests and dispositions. Arrests or charges that had been adjudicated at the time of 


4Shein, Marcia. “At What Age Can You Be Tried as an Adult in Georgia?” Federal Criminal Law Center, 2014. 
Available at https://federalcriminallawcenter.com/2014/1 1/age-can-tried-adult-georgia/ Accessed October 11, 
2019. 

Certain violent crimes include murder, murder in the second degree, voluntary manslaughter, rape, aggravated 
sodomy, aggravated child molestation, aggravated sexual battery, and armed robbery with a firearm. Children 
younger than 17 may also be prosecuted for these offenses in juvenile court, at the discretion of the district 
attorney or superior court (GA Code § 15-11-560 [2014]). 

National Juvenile Defender Center. “Michigan Juvenile Indigent Defense Delivery System.” July 2018. Available 
at https://njdc.info/practice-policy-resources/state-profiles/michigan/. Accessed October 11, 2019. 

°7Epps, Quaniqua. “Change in SC Law Has 17-Year-Olds Considered Juveniles.” WCBD News 2, November 3, 
2017. Available at https://www.counton2.com/news/south-carolina-news/change-in-sc-law-has-17-year-olds- 
considered-juveniles/. Accessed October 11, 2019. 


8 Certain felony charges include any felony that carries a sentence of up to 15 years, assault and battery of a high 
and aggravated nature, or any felony that carries a sentence of up to 10 years for youth with two or more prior 
convictions (South Carolina Code of Laws, Section 63-19-1210 20-7-7605). 
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data production but did not result in a conviction were not included. Charge categories were 
provided for each conviction, but no charge was provided for arrests pending disposition. 
Data were provided through November 2019. 


e South Carolina. South Carolina records include data from the South Carolina State Law 
Enforcement criminal history files. Data were obtained in six files: identification, arrest, 
count, custody, judicial, and aliases. Arrest, count, and judicial files were combined using 
encrypted record-level linking IDs and unique internal linking numbers. Information on 
charges included all South Carolina charge codes and literal descriptions. Data were provided 
through June 2019. 


NATIONAL STUDENT CLEARINGHOUSE DATA 


The NSC maintains a comprehensive administrative database with information on student 
enrollment in postsecondary education and postsecondary degree completion. Participating 
postsecondary institutions report information on enrollment and degree attainment directly to the 
NSC. This includes data on more than 99 percent of U.S. colleges and universities, including all 
three JC partner institutions. The NSC shares this information with outside organizations through 
its StudentTracker service.’ A record containing the following elements is provided for each 
enrollment spell at each institution in which a student enrolled: 


e Name of the postsecondary institution attended 
e Dates of enrollment, enrollment status (for example, full-time, part-time) 
e Program major 
e Information on whether a student graduates from the program 
e For graduates: 

- Graduation date 

- Type of degree 

- Degree major 
We gave the NSC the names, dates of birth, and Social Security numbers for participants who 
provided consent to participate in the study. The NSC matched these records to its database using 
a proprietary algorithm that accounts for variations in names and dates of birth. To ensure we 
matched as many records as possible, we submitted multiple spellings or variants of names and 


dates of birth when they differed across our data collections, then reconciled the matches from 
NSC. The StudentTracker report from NSC included postsecondary enrollment information for 


2° National Student Clearinghouse, “StudentTracker.” Available at 
https://www.studentclearinghouse.org/colleges/studenttracker/ 
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participants beginning in January 1, 2015, before any cohorts of participants would have enrolled 
in YC, through October 16, 2019, when we submitted our request to NSC. 


Using the StudentTracker report, we created indicator variables for each enrollment spell or 
credential record reported for a participant. Based on each participant’s YC cohort, we created 
indicators for whether the participant was enrolled during four time points: within six months of 
completing YC, one year following completion of YC, two years following completing of YC, 
and at any point after YC completion.*° Enrollment at a point in time was defined as whether the 
participant was enrolled in the semester in which that point in time occurred. We also created 
indicators of whether a credential record occurred within six months of YC completion and at 
any point after YC completion. For each enrollment spell, we also created indicators for whether 
the participant was enrolled in a two- or four-year institution and whether the participant was 
enrolled in a private institution. 


FOLLOW-UP SURVEY 


1. Collection method 

The sample for the follow-up web survey included JC participants in Cohorts 4, 5 and 6 who 
gave consent to participate in the evaluation (and whose parents or guardians did so, when 
necessary). The follow-up survey covered five broad topics: 


e Experiences during JC, such as the services the participant received 


e Employment and earnings, such as the characteristics of a current job or the participant’s 
recent work search efforts 


e Educational experiences, including attainment and future plans 


e Involvement in the court system, such as whether the participant was arrested and convicted 
of a crime 


e Views about the value of different aspects of the JC program 


Participants were notified about the survey request via mail, email, and text message using a 
two-phase approach. This approach was designed to address challenges related to aging contact 
information, which was originally provided as part of the BIF. By the time the follow-up survey 
was collected, contact information from the BIF was up to two years old. This was particularly 
problematic for Cohort 4, which was fielded later than planned due to delays in receiving 
approval from the Office of Management and Budget for the follow-up data collection. 


3° Because we could only observe enrollment outcomes through October 2019, we did not include outcomes at two- 
years following YC for YC Cohort 6 in the analysis. These outcomes would have only included a five-month 
period from June to October 2019, when the NSC data were collected. 
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Phase 1. In the first phase, participants were directed to the web survey via email and text 
message. The first phase did not use home addresses because we expected that much of the 
address information from the BIF may have changed as participants left the JC program. 


Phase 2. In the second phase, conducted about two to three weeks into data collection, trained 
locators at Mathematica’s Survey Operations Center attempted to collect additional contact 
information for participants who had not responded. To do this, they reached out to sites for any 
additional contact information. In addition, the full sample was submitted to Accurint*!, and all 
possible telephone numbers, home addresses, and email addresses were returned to locators for 
review and confirmation with the participants. The locators reviewed contact information for 
participants who had not completed and made calls to all possible telephone numbers and 
alternate contacts collected in the BIF. The purpose of these calls was to alert participants of the 
survey, answer any questions they might have, and confirm the best information for resending 
the web survey information, if needed. In this phase, the study also created an Instagram account 
and used Instagram direct messaging to contact participants who had provided their Instagram 
contact information in the BIF. 


All participants who completed the survey received $30 in Amazon.com credit. 


2. Timing 
We initiated the follow-up survey 18 to 20 months after the start of each JC program (Exhibit 
B.2). Each survey was fielded for approximately three months. 


Exhibit B.2. Follow-up survey collection dates 


JC cohort Program start date ae) | Cop En 0] oM-LULai-nYarere) |(-Yead (ol ae l-1K-) 
4 July 2017 March 2019-—June 2019 
5 January 2018 September 2019-—December 2019 
6 July 2018 October 2019-—January 2020 


3. Response rates 

Despite our extensive efforts to collect surveys from all consenting participants, only 47 percent 
completed the follow-up survey (Exhibit B.3). As discussed above, we believe this was largely 
due to outdated contact information. However, it was generally not possible to distinguish 
between youth who had not received the follow-up survey and youth who ignored it; email and 
Instagram accounts might have been dormant but not canceled, phone messages might not have 
indicated that the participant was no longer at that number, and mail might not have been 
properly forwarded or returned to the sender. Only two participants directly refused to answer 


31 Accurint is a LexisNexis product which allows users to retrieve data from their national database of public 
records including U.S. addresses and phone numbers. 


100 


Appendix B Mathematica 


the survey. For more information on the characteristics of responders relative to nonresponders, 
see Appendix C. 


Exhibit B.3. Response rates 


Consenting Number of surveys 
participants fexey any eo) (K-19 | Percentage complete 
ss) 47 

Georgia 44 21 48 
Michigan 36 17 47 
South Carolina 36 9 25 
Cohort 5 
Georgia 37 16 43 
Michigan A 24 59 
South Carolina 34 11 32 
Cohort 6 104 52 50 
Georgia 39 17 44 
Michigan 44 26 59 
South Carolina 21 9 43 
Total KRY 150 45 


Source: BIF and follow-up survey data. 


Notes: Analysis sample includes JC participants in JC Cohorts 4-6 who completed a BIF. Response 
rates are estimated as the total number of follow-up survey respondents divided by the total 
number of participants who completed a BIF. 


TEXT MESSAGE SURVEY 


JC participants from Cohorts 5 and 6, who provided prior consent to participate in the evaluation 
and permission to contact them via text message, were asked to complete a brief survey 
administered by text messaging.*” The survey was conducted on a monthly basis for six months, 
beginning 10 to 13 months after the participant started the JC program (Exhibit B.4). The brief 
survey was designed to provide snapshots of the progression over time the respondents make in 
their employment, earnings, and education between the end of the Job ChalleNGe program and 
the follow-up survey. For each round of collecting monthly text message data, each participant 
was asked to answer three to five questions tailored based on his or her circumstances. For 
example, participants who responded to the first question that they are not currently working did 
not receive the follow-up questions about hours worked and hourly pay rate at a job. During the 
fifth administration of the monthly text message survey, the study team also used the survey to 
collect updated contact information from the participants to support the follow-up survey data 
collection effort. 


32 Fifty-four percent of study participants in Cohorts 5 and 6 consented to receiving the text message survey. 
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Exhibit B.4. Dates of text message survey collection 


JC cohort Program start date Dates of text message survey collection 
5 January 2018 March 2019-—September 2019 
6 July 2018 May 2019—November 2019 


Participants eligible for the text message survey were notified of this data collection effort via 
mail, email, and text message. The mail and email outreach were added to improve legitimacy of 
the text message they received. Text messages were sent out in bulk each month using a texting 
vendor. Participants received the text messages from a number with an area code that was local 
to their JC program site. Participants had approximately two weeks to respond to the questions 
each month and received reminder prompts at various intervals during that window if they had 
not yet completed all questions. Participants were also prompted with an opportunity to opt out 
of all future text messages if they did not wish to continue participation. Participants who 
completed the survey received a $3 in Amazon.com credit each month they completed the text 
message survey. 


Approximately 11 percent of the youth who agreed to be contacted by text completed all six 
rounds of the text survey (Exhibit B.5). Thirty percent of those in Cohort 6 and 27 percent of 
those in Cohort 5 responded to at least one round of the survey. As mentioned previously, the 
text survey protocol did allow participants to opt out in each round of the survey. Of the cases 
that did not complete any month of the text survey, 52 percent were opt-outs, and most of those 
cases had opted out during the first month of fielding. 


Although we know which cases opted out as part of the text survey protocol, we expect that the 
remainder of nonresponse was due to participants blocking the messages manually through their 
texting application, participants ignoring messages, or the use of telephone numbers that might 
have changed since fielding the BIFs. In each round, before the start of survey questions, the text 
survey protocol asked the participant to confirm that we were contacting the correct number. In 
the first month of the survey, we received responses from 10 telephone numbers indicating the 
number was incorrect. Another 13 participants indicated the telephone number was correct but 
did not complete the survey questions that followed. 


Exhibit B.5. Response rates of text message surveys 


Number of survey months completed 


3 
JC Total 


Coxe) se) a Mammy- 111) ©) (<n | 


Note: Includes all participants who consented to receive the text message survey. 
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Adjustments for Nonconsent and Nonresponse 
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The purpose of this study is to provide insights about the experiences of all YC and JC youth 
under the JC grants. However, for most of the quantitative analyses, data are limited to the set of 
youth who consented to participate in this study and, in the case of survey data, responded to the 
relevant survey. This appendix discusses considerations in interpreting the study results on 
participant characteristics, services received, and post-program outcomes as representative of all 
YC and JC youth. We also describe the adjustments we made to analyses to increase the 
generalizability of results to a broader set of youth than those for which we have data. 


A. Obtaining consent and collecting background information 


All analyses in this study using survey or administrative data cover only the YC participants who 
consented to participate in the study. Although we collected partial data on earlier cohorts from 
the pilot, we collected data in full for Cohorts 4—6. Over the course of YC or at JC entrance, 
youth received information on the study and were asked to participate in it. Youth age 18 and 
older were asked to sign a form consenting to participate. Youth younger than 18 were asked to 
provide their assent for participation, and parental consent was also required. As is common in 
consent-based studies, not all YC youth consented to participate in the study. For instance, some 
youth chose not to participate due to privacy concerns. Others faced logistical issues, such as 
difficulty accessing their parents for consent (for youth younger than 18). Thus, this study does 
not contain all youth who were eligible for study inclusion. 


After obtaining consent (and assent, if needed) for each participant, the study team asked cadets 
to complete a background information form that collected key information on participants. 
Nearly all of the cadets who consented to participate in the study also completed this form. 
Analyses are limited to the set of participants who both consented and completed a background 
information form. For the purposes of this appendix, we refer to nonconsenters as anyone for 
whom we did not obtain consent or who did not complete the background information form. 


Overall, 63 percent of YC cadets consented to participate in the study (Exhibit C.1). Consent 
rates were 15 percentage points higher for JC participants than for participants in the YC 
program only. This difference was driven by the timing of data collection, the nature of data- 
collection efforts for each group, and potentially by the different nature of participants in the two 
programs. JC consent rates were also lower in Georgia than they were in Michigan and South 
Carolina. This was likely because some participants in the Georgia JC program came from 
another YC site, which was not a JC grantee, and therefore we did not obtain consent and collect 
baseline data from these participants during the YC program. 
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Exhibit C.1. Consent rates (reported in percentages) 


Program ae} r-)) .eT=Yo} Ke] =) WM Efe alte F-Ta) South Carolina? 
Total 67 63 70 71 
YC only 63 63 61 61 
Jc 78 64 89 86 
Sample size 1,460 775 405 280 


Source: Program administrative data and background information form. 


Notes: Analysis sample includes participants in YC Cohorts 4—6. Consent rates estimated as the total 
number of study participants divided by the total number of program participants. 


aSouth Carolina consenting participants and total participants are limited to youth who completed YC. 


1. Comparison of consenters and nonconsenters 


The goal of this study is to assess implementation and outcomes for all youth in the YC and JC 
programs under the JC grants. Nonconsent will lead to a biased analysis if consenters are not a 
random subset of the eligible study population and are instead somehow systematically included 
and are nonrepresentative of the group as a whole. In this case, our analysis would describe the 
experiences of consenters, but bias would limit the ability to generalize the results to the full 
population of youth. To assess the extent of potential bias in the sample due to nonconsent, we 
compared the observable characteristics of consenters and nonconsenters. To do this, we used 
program data the sites provided to identify the full set of youth that we would have ideally liked 
to include in the analysis. We compared the characteristics of consenters and nonconsenters 
using program data on each youth participant’s cohort, age, race/ethnicity, and gender. Michigan 
and South Carolina programs also provided data on each cadet’s pre-program court involvement, 
which we included in this analysis. 


We matched consenting cadets from our study sample to the program data. 


e For Michigan and South Carolina, we matched cadets based on name, date of birth, gender, 
and Social Security number (when available). To account for inconsistencies across data 
sources, we matched participants using a probabilistic matching algorithm. Any consenting 
participant who was not matched to the program data using this matching algorithm was then 
manually compared to program data to identify a match. 


e Because South Carolina only provided program data for YC cadets who completed the 
program, unmatched South Carolina cadets were assumed to have not completed the program 
and were not included in estimates of consent rates. For Georgia, program data did not 
include any personally identifiable information for matching to the consent sample. Georgia 
cadets were therefore matched on the characteristics available in both the program data and 
the background information form: cohort, age, race/ethnicity, and gender. For example, if the 
Georgia program data noted 10 cadets in Cohort 4 who were White, 17-year-old females but 
data from the background information form noted only 5 cadets with these characteristics, we 
assumed a 50 percent consent rate among White, 17-year-old females in Georgia Cohort 4. 
Of the 488 total consenters in Georgia, 71 could not be directly matched to the program data 
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using data from their background information forms.** This is likely due to inconsistent 
reporting across data sets and missing information.** Therefore, the consent rate for Georgia 
participants by characteristics is lower than the true Georgia consent rate, as presented in 
Exhibit C.1. 


Consistent with systematic nonconsent, we see meaningful and statistically significant 
differences between consenting and nonconsenting participants (Exhibit C.2). Consenters were 
12 percentage points more likely than nonconsenters to be 16 years old and 11 percentage points 
less likely to be 18 years old. Consenting participants were also 5 percentage points more likely 
than nonconsenters to be White and 8 percentage points less likely to be Black. No statistically 
significant differences were evident in the gender or court involvement of consenters and 
nonconsenters. To statistically test whether the differences between consenters and 
nonconsenters were significant across all characteristics, we also performed a logistic regression 
of consent on the full set of youth characteristics. The results showed a clear difference between 
the characteristics of the two groups, which was statistically significant at the .01 level. 


Exhibit C.3 shows the characteristics of consenters and nonconsenters at each site. Overall, the 
patterns found in each individual site were mostly the same as those found across the three sites 
combined, with some exceptions. In Michigan, consenters were a statistically significant 11 
percentage points less likely than nonconsenters to be male, and in Georgia, consenters were 
more likely to be Black and less likely to be White. In Georgia, consent rates by cohort varied 
more than in the other two sites. In South Carolina, no statistically significant differences were 
evident between consenters and nonconsenters, but this might also have been due to smaller 
sample sizes. 


33 This includes both (1) participants who completed the baseline information form but for whom no one in the 
program data had a matching set of characteristics and (2) participants with a set of characteristics that were more 
common among consenters than among all cadets in the program data. 

34 Unmatched participants are more likely to be identified as Hispanic or other/mixed race ethnicity, suggesting that 
discrepancies may be because of inconsistent categorization of race and ethnicity across data sets. 
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Exhibit C.2. Baseline characteristics of consenters and nonconsenters across all sites 
(reported in percentages) 


Characteristic Consenters Nonconsenters p-value? 
Age at YC entrance wee 
16 55 43 
17 36 37 
18 9 20 
Male 80 82 
Race and ethnicity = 
Hispanic 4 2 
Non-Hispanic, Black 52 60 
Non-Hispanic, White 38 33 
Non-Hispanic, other race 6 5 
Court-involved® 26 30 
Cohort ie 
4 39 29 
5 32 41 
6 29 30 
Sample size 897 563 


Source: Program administrative data and background information form. 
Note: Analysis sample includes participants in YC Cohorts 4-6. 


aWe conducted chi-squared tests to assess the differences between consenting and nonconsenting 
youth. * p< .1; ** p < .05; *** p< .01. 


bMichigan and South Carolina only. 


2. Weighting for nonconsent 


To mitigate the effects of the systematic nonconsent on the analysis and the bias it might cause, 
we reweighted all data such that the observable characteristics of the analysis sample match the 
observable characteristics of the full eligible population.*> To do this, we assigned a consent 
weight to each consenting participant in our data. We applied these weights for all analyses using 
the background information form, data from the National Student Clearinghouse (NSC), and 
criminal justice administrative data. To obtain NSC and criminal justice data, we provided a list 
of consenting participants’ personal information to data providers for matching. Therefore, the 
response rates for these data sets are estimated from the share of participants who consented and 
for whom we have adequate data for matching. We interpret any unmatched individuals as 


35 Tn South Carolina, this requires the additional assumption that completers and noncompleters do not differ on 
observable characteristics. 
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Exhibit C.3. Baseline characteristics of consenters and nonconsenters, by site (reported in percentages) 


fCT-Yo) ce] F-) Michigan South Carolina 


Non- p- Non- p- Coy a 
Characteristic Consenters consenters value? Consenters consenters value? Consenters consenters p-value? 
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Source: Program administrative data and background information form. 
Note: Analysis sample includes participants in YC Cohorts 4-6. 
aWe conducted chi-squared tests to assess the differences between consenting and nonconsenting youth. 


Michigan and South Carolina only. Data which was not available is indicated with a “-. 
*p<.1;** p< .05; *** p< .01. 
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having not attended school (for NSC data) or interacted with the criminal justice system (for the 
justice data). Hence, we assumed a 100 percent match rate between the study’s list of consenting 
participants and each of these two sources of administrative data. 


To estimate consent weights, we assigned each participant an estimate, » , the probability of 
consent for individual 7. To estimate » , we performed a logistic regression of consent on the 


observable characteristics: age, gender, race/ethnicity, court involvement (where available), 
cohort, and site. We used a fully saturated model, which means that we included indicator variables 
for each value of each characteristic and an indicator signaling that information about that 
characteristic for a participant is missing from the data. We estimated the model on the group of 
participants that is in the program data using the same sample we used to create the earlier exhibits 
in this appendix. We estimated the predicted probability of consent, p,, as the predicted value 


from the logistic regression.*° For each consenting youth participant, i, we then assigned a sample 
consent weight, w,aS w, =1/ p,- 


We used these estimated consent weights, w,, to reweight the study sample to reflect the 


observable characteristics of the full relevant participant population for all analyses based on the 
background information form, NSC data, and criminal justice administrative data. 


B. Follow-up survey 


We collected follow-up survey data for each youth participant about 18 to 20 months after the 
start of the JC program to provide additional information on the experiences of JC participants 
and on their education, employment, and criminal justice outcomes. We sent the survey by text 
message, email, and mail to participants who consented to participate in the study. The study 
team also created an Instagram account and contacted participants through that platform. We also 
contacted study participants who did not respond to the survey by phone. 


Although we made extensive efforts to maximize survey response rates, only 44 percent of 
consenting participants completed the follow-up survey (Exhibit C.4). This was partially due to 
an inability to contact youth. Some participants did not provide contact information when they 
were completing the baseline information form, limiting our ability to contact them when they 
were due to be invited to complete their follow-up surveys. Additionally, because the follow-up 
survey was collected more than a year following the start of the JC program, the information on 
how and where to contact the youth participant was often outdated. It was often impossible to 
distinguish between youth who had not received the follow-up survey and youth who ignored 
it—email and Instagram accounts might have been dormant but not canceled, voicemail 
recordings might not have indicated that the participant was no longer at that number, and mail 
might not have been properly forwarded to the participant or returned to us. Wherever possible, 


36 For some analyses, we include data on JC participants in Cohorts 4—6 who participated in YC Cohort 3. For these 
youth, predicted values were assigned using Cohort 4 predicted values because we did not collect data on 
participants in YC Cohort 3 who did not participate in the JC program. 
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we attempted to obtain updated contact information from sites, but in many cases this 
information was also outdated by the time of the follow-up survey. Only two participants whom 
we affirmatively contacted refused to complete the survey. The response rate was highest among 
Michigan participants and lowest among South Carolina participants. 


Exhibit C.4. Response rates (reported in percentages) 


Program Be} F-1) eT-Yo) ce] f=) Michigan South Carolina 


Total 45 45 55 32 
Sample size 332 120 121 91 


Source: Background information form and follow-up survey data. 


Notes: Analysis sample includes JC participants in YC Cohorts 4—6 and JC Cohorts 4-6 who completed 
a background information form. Response rates are estimated as the total number of follow-up 
survey respondents divided by the total number of participants who completed a background 
information form. 


1. Comparison of survey responders and nonresponders 


As with nonconsent to participate in the study, nonresponse to the follow-up survey will lead to 
biased results if youth who responded to the survey are systematically different from those who 
did not respond to the survey. This could happen if, for example, participants who had a positive 
experience in the JC program or who were employed were more likely to respond to the survey. 
Although we do not have survey-based data upon which to explore differences in outcomes 
between survey responders and nonresponders, we can assess whether the characteristics of 
responders are similar to those of nonresponders using information from the background 
information form. Because all consenters completed the background information form, we can 
compare survey responders to nonresponders using a larger set of characteristics than we could 
use for the consent analysis. 


Exhibits C.5 to C.7 show the demographic and household, education and employment, and self- 
reported delinquent behavior characteristics of survey responders relative to nonresponders, 
weighted using nonconsent weights. Across all measured characteristics, the only difference in 
characteristics that was statistically significant at the .05 level was in employment history. 
Follow-up survey responders were 11 percentage points less likely than nonresponders to have 
been employed directly before the YC program and 9 percentage points less likely to have ever 
had a paying job for three months or more. Follow-up survey responders were less likely than 
nonresponders to be in Cohort 4 and more likely to be in Cohorts 5 and 6, although this 
difference was only significant at the .1 level. 


To test whether systematic differences existed between responders and nonresponders across all 
participant characteristics, we performed a logistic regression of response on a subset of 
available covariates representing key characteristics: age, gender, race/ethnicity, cohort, court 
involvement, history of at least three months of employment, and educational attainment (at least 
10th grade and at least 11th grade). The results (not shown) indicate a difference between 
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responders and nonresponders that was statistically significant at the .1 level but not the .05 
level. 


Exhibit C.5. Baseline demographic characteristics of JC youth who responded and did 
not respond to the follow-up survey (reported in percentages) 


Characteristic BIF and FUS BIF-only P-value? 
responders ie=¥-) ofe) ale (=) c=) 
Age 
16 57 54 
17 30 36 
18 13 10 
Male 80 74 


Race and ethnicity 


Hispanic 7 10 
Non-Hispanic, Black 36 43 
Non-Hispanic, White 43 37 
Non-Hispanic, other race 13 10 
Foster care involvement 0 2 

Free or reduced-price lunch status°® 65 67 

Unstable housing? 5 7 

Ever received special education services 21 19 

Married 1 0 

Has a child 1 1 

Cohort * 
4 30 4 
5 39 35 
6 31 24 
Sample size 150 182 


Source: Background information form (BIF) and follow-up survey (FUS). 
Notes: Analysis sample includes participants in JC Cohorts 4—6 who completed the BIF. 


4Statistical significance is estimated using chi-squared difference tests to compare differences between 
the BIF and FUS responders and the BIF-only responders. * p< .1; ** p< .05; *** p< .01. 


>Foster care involvement and housing status were self-reported at the time of the BIF collection. 


‘Free and reduced-price lunch status was self-reported based on the two years prior to background 
information form collection. 
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Exhibit C.6. Baseline education and employment characteristics of JC youth who 
responded and did not respond to the follow-up survey (reported in percentages) 


BIF and FUS BIF-only 


Characteristic ie=¥-y ofe) ale l=¥ g-) responders 


Educational attainment 


8th grade or below 5 7 


10th grade 42 42 


12th grade 11 7 


Ever suspended 73 76 


Employed directly before YC program 22 33 


Sample size 150 182 


Source: Background information form (BIF) and follow-up survey (FUS). 


Notes: Analysis sample includes JC participants in YC Cohorts 4-6 and JC Cohorts 4-6 who completed 
the BIF. 


aStatistical significance is estimated using chi-squared difference tests to compare differences between 
the BIF and FUS responders and the BIF-only responders. 


*p<.1;** p< .05; *** p< .01. 
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Exhibit C.7. Self-reported delinquent behavior and justice system involvement of JC 
youth who responded and did not respond to the follow-up survey (reported 
percentages) 


BIF and FUS BIF-only 
Characteristic responders responders p-valuea 
Used marijuana in past six months 42 47 
Used another drug in past six months 22 21 
Any court involvement 36 36 
Ever arrested or taken into custody 28 25 
Any status offense> 14 16 
Ever convicted 15 13 
Ever detained in a juvenile facility 9 12 
Ever detained in an adult facility 7 5 
On probation or parole at YC entrance 10 13 
Sample size 150 182 


Source: Background information form (BIF) and follow-up survey (FUS). 


Notes: Analysis sample includes JC participants in YC Cohorts 4-6 and JC Cohorts 4—6 who completed 
the BIF. 


aStatistical significance is estimated using chi-squared difference tests to compare differences between 
the BIF and FUS responders and the BIF-only responders. * p < .1; ** p< .05; *** p< .01. 


bA status defense is defined as a noncriminal act that is a violation of the law for minors. Examples 
include running away from home and underage use of alcohol. 


2. Weighting 


Although the characteristics of survey responders and nonresponders were similar on many 
dimensions, we weighted follow-up survey data used in the outcomes analyses to account for 
survey nonresponse. We used a two-level weighting method to account for both nonconsent and 
survey nonresponse. This is designed to reweight the follow-up survey responders to have the 
same observable characteristics as the full eligible study population. 


To generate weights for follow-up survey respondents, we first estimated 7,, the probability that 


JC participant i responded to the survey conditional on having consented to participate. To 
estimate r,, we performed a logistic regression of a binary variable indicating survey response on 


observable baseline characteristics: age, gender, race/ethnicity, court involvement, history of at 
least three months of employment, and educational attainment. For educational attainment, we 
included two indicator variables signaling whether the participant had completed 10th grade and 
whether the participant had completed 11th grade. For all other variables, we used a fully 
saturated model, which means that we included indicator variables for each value of each 
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characteristic and an indicator signaling that information about that characteristic for the youth is 
missing from the data. The model was estimated on the group of consenting participants only, 
because they form the group of youth that we included in the follow-up survey fielding effort. 


We then generated weights by estimating the overall probability of inclusion in the follow-up 
survey as p,xr, , where p, is the probability of consenting to being in the study as described in 


Section A of this appendix. We estimated the nonresponse weight as w, ; =1/ ( Pp; * r.) . To ensure 


that results are not driven by a few outliers, we used a trimming method to cap weights. We set 
the maximum weight as three times the median weight value (Van de Kerckhove et al. 2014). 
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Differences in post-program outcomes between JC and YC-only participants might be driven by 
differences between the characteristics of participants in each group. To provide insights about 
the extent to which this might be true, we performed an ordinary least squares regression analysis 
to estimate the differences in outcomes between the two groups after controlling for observable 
characteristics. It is important to note that this analysis only controls for observable 
characteristics and not the full set of differences between JC and YC-only participants. 
Therefore, the difference that remains after taking into account the observed demographic 
characteristics cannot be interpreted as solely due to JC participation. For example, if—even 
holding constant all observable characteristics—youth who are more ambitious are more likely to 
sign up for JC then we may observe that JC youth have higher earnings. Therefore, if we observe 
a positive association between JC participation and outcomes, that difference might have existed 
even in the absence of JC so therefore does not represent the effect of JC. 


We estimated the following regression model of outcomes on JC participation: 
(1) Y,=Byot+ BIC, +X, +€;> 


where y, is the outcome measure based on administrative data (a measure of educational 


engagement or contact with the justice system) for each participant i and JC is an indicator 
variable taking the value of 1 for YC participants who participate in JC and 0 for YC participants 
who do not. X is a vector of covariates from the background information form, including the 
following characteristics: gender, age, race, YC cohort, court involvement before YC, 
educational attainment before YC, free or reduced price-lunch receipt before YC, history of 
special education, stable housing before YC, site, and YC cohort. ¢; is an individual-specific 
error term. We used a fully saturated model, which means that we included indicator variables 
for each value of each characteristic and an indicator signaling that information about that 
characteristic for a YC participant is missing from the data. In this model, /, is the mean 


difference in the outcome measure between JC participants and YC participants after taking into 
account differences across youth in the other characteristics included in the model. 


Exhibit D.1 shows the results of the regressions of educational outcomes on JC participation, 
controlling for observable characteristics of the youth. Exhibit D.2 shows similar regression 
results for criminal justice outcomes. 
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Exhibit D.1. Regression-adjusted differences between educational outcomes of JC youth 
and YC-only youth 


WF-Tat-10) (=) Coefficient Standard error p-value 
Enrollment in school 


One year following YC 0.01 0.021 


Enrolled in two-year college 


One year following YC 0.02 0.020 


Enrolled in four-year college 


One year following YC -0.01 0.007 


Enrolled in private institution 


One year following YC 0.00 0.004 


Obtained certification” 


Any time following YC°¢ 0.42 0.037 aes 


Source: National Student Clearinghouse sample weighted data (N = 984). 
Notes: Analysis sample includes participants in YC Cohorts 4-6. 
4@Outcomes calculated two years following YC do not include YC Cohort 6. 


>Degree data were not available for Michigan participants. Sample is limited to Georgia and South 
Carolina. 


°Data collected through October 2019. 
*p<.1;** p< .05; *** p< .01. 
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Exhibit D.2. Regression-adjusted differences between criminal justice outcomes of JC 
youth and YC-only youth 


NVéeTarele) (=) Coefficient Standard error p-value 
Arrested for a new offense 


Any time following YC -0.06 0.027 i 


One year following YC -0.01 0.014 


Drug offense 


fo) 
oO 
= 


Any time following YC 0.006 


One year following YC 0.00 0.006 


Property crime 


Any time following YC 0.00 0.011 


One year following YC 0.00 0.009 


Source: Criminal justice weighted administrative data (N = 984). 

Note: Analysis sample includes participants in YC Cohorts 4—6. Criminal justice involvement rates 
represent the share of participants with a criminal justice event any time following the YC 
program. YC program end dates were considered to be May 31, 2017, for Cohort 4; December 
31, 2018, for Cohort 5; and May 31, 2018, for Cohort 6. Criminal justice data were collected 
through June 2019 for South Carolina, November 2019 for Michigan, and December 2019 for 
Georgia. 

*p<.1;** p< .05; ** p< .01. 
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For the primary analyses in this study, we present results pooled across all three grantees. 
However, the experiences of youth in the YC and JC programs were not uniform. The grantees 
differed in participant characteristics, program components, staff characteristics, and local 
economic conditions. In this appendix, we present the results of key analyses by grantee. 


Exhibit E.1. Baseline characteristics of YC youth, by grantee (reported in percentages) 
(pooled results in Exhibit II.2) 


Characteristic (eT=Yo) ce] =) WY ited alte Feta) South Carolina 


Age 


—_ 
N 
oo 
© 
io) 
o 
© 
KR 


= 
o 
O 
co 
I) 
N 
o 
~N“N 
o 


© 
—_ 
No 
K 


Hispanic 


NO 
oO 
ep) 
oO 
wo 
= 


Non-Hispanic, White 


= 
1) 
= 


Foster care involvement? 


Housing status? 


nN 
‘ 
nN 


Unstable housing 


= 
oO 
oO 


Married 


Sample size 488 282 214 


Source: Background information form weighted data. 
Note: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 


@Foster care involvement and housing status were self-reported at the time of the background information 
form collection. 


>Free lunch status self-reported based on the two years prior to background information form collection. 
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Exhibit E.2. Baseline education and employment characteristics of YC youth, by 
grantee (reported in percentages) (pooled results in Exhibit I1.3) 


Characteristic LeT-Yo) ce] =) Michigan South Carolina 


Educational attainment 


8th grade or below 10 5 4 


10th grade 39 41 41 


12th grade 4 5 3 


Ever suspended 78 78 88 


Employed directly before YC program 28 28 29 


Sample size 488 282 214 


Source: Background information form weighted data. 
Note: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 
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Exhibit E.3. Self-reported delinquent behavior and justice system involvement of YC 
youth at the time of enrollment, by grantee (reported in percentages) (pooled results in 
Exhibit II.4) 


Characteristic Georgia Michigan South Carolina 
Used marijuana in past six months 54 58 47 
Used another drug in past six months 23 27 18 
Any court involvement 43 37 39 
Ever arrested or taken into custody 33 24 26 
Any status offense? 18 22 17 
Ever convicted 19 17 20 
Ever detained in a juvenile facility 19 11 16 
Ever detained in an adult facility 7 3 5 
On probation or parole at YC entrance 15 12 14 
Sample size 488 282 214 


Source: Background information form weighted data. 
Notes: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 


aA status defense is defined as a noncriminal act that is a violation of the law for minors. Examples 
include running away from home and underage use of alcohol. 
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Exhibit E.4. Baseline characteristics of YC youth overall and by JC participation (reported 
in percentages) (pooled results in Exhibit III.2) 


Georgia WM ifedal te FT) South Carolina 

Characteristic JC YC only JC YC only JC 
Age 

16 53 40 57 61 53 51 

17 36 43 38 25 34 32 

18 11 16 5 14 12 16 
Male 83 78 77 75 81 76 
Race and ethnicity 

Hispanic 8 9 13 11 5 2 

Non-Hispanic, Black 64 63 18 16 59 44 

Non-Hispanic, White 19 21 59 62 25 38 

Non-Hispanic, other race 9 7 10 11 10 16 
Foster care involvement? 1 2 2 1 1 1 
Free and reduced-price lunch status> 79 73 61 59 75 70 
Housing status? 

Stable housing 96 94 97 95 97 95 

Unstable housing 4 6 3 5 3 5 
Ever received special education 15 14 23 27 27 22 
services 
Married 1 0 0 0 0 1 
Has a child 6 0 3 0 3 1 


Educational attainment 
Last grade completed in school 


8th grade or below 11 6 6 5 2 7 
9th grade 28 20 25 26 25 22 
10th grade 38 44 41 41 42 40 
11th grade 19 25 23 22 27 27 
12th grade 4 4 4 7 3 4 
High school diploma or GED 2 6 1 2 1 0 
Employment 80 69 85 67 87 90 
Employed directly before YC 28 28 26 31 27 32 
program 
Ever hada paying job for3ormore 41 44 47 46 44 52 
months 


Source: Background information form weighted data. 
Note: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 


4Foster care involvement and housing status self-reported at the time of the background information form 
collection. 


>Free lunch status self-reported based on the two years prior to background information form collection. 
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Exhibit E.5. Self-reported delinquent behavior and justice system involvement of YC 
youth at the time of enrollment overall and by JC participation (reported in percentages) 
(pooled results in Exhibit III.3) 


T-Yo} ge] F-] Wied ate FTA) South Carolina 
Characteristic 
Used marijuana in past six months 58 40 62 51 51 42 
Used another drug in past six 25 15 28 27 17 19 
months 
Any court involvement 45 37 41 33 40 38 
Ever arrested or taken into 33 32 24 24 27 25 
custody 
Any status offense? 20 11 25 19 18 16 
Ever convicted 20 14 21 12 21 19 
Ever detained in a juvenile facility 1 12 12 10 19 13 
Ever detained in an adult facility 6 9 2 4 5 6 
On probation or parole at YC 15 13 12 11 15 12 
entrance 
Sample size 383 105 174 108 123 91 


Source: Background information form weighted data 


Notes: Analysis includes all youth in YC Cohorts 4-6 who completed a background information form. 
Youth were categorized as having court involvement if at baseline they reported ever being 
arrested, found guilty of a status offense, convicted of a crime, spent time in a juvenile or adult 
detention facility, or if they were on probation or parole at the time of entering YC. 


aA status defense is defined as a noncriminal act that is a violation of the law for minors. Examples 
include running away from home and underage use of alcohol. 


Exhibit E.6. Employment outcomes at follow-up for JC participants (reported in 
percentages unless otherwise noted) (pooled results in Exhibit VI.5) 


Employment outcomes eT-Yo} xe F-] Michigan South Carolina 
Employed following program 

At JC exit 47 59 42 

At follow-up survey 83 79 79 
Average weekly earnings (dollars) 378 423 317 
Average hours per week (hours)? 31 34 33 
Sample size 54 67 29 


Source: Follow-up survey weighted data. The follow-up survey was administered 16 to 23 months after 
youth started the JC program. 


Notes: Analysis sample includes JC participants in Cohorts 4-6. 


aAverage earnings and hours are calculated across all participants. Participants who were not currently 
employed were assigned a value of zero for both earnings and hours. 
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Exhibit E.7. Employment characteristics among those employed at follow-up (pooled 
results in Exhibit VI.6) 


Employment outcomes eT-Yo} ce] F-] Michigan South Carolina 
Average weekly earnings (dollars) 455 537 403 
Average hours per week (hours) 38 44 42 
Job provides fringe benefits (percentage) 72 61 70 
Average job tenure (weeks) 8 9 10 
Sample size 46 51 22 


Source: Follow-up survey weighted data. The follow-up survey was administered 16 to 23 months after 
youth started the JC program. 

Notes: Analysis sample includes JC participants in Cohorts 4-6 who reported being employed at the 
time of the follow-up survey. 


Exhibit E.8. Education outcomes for JC and YC only participants (reported in 
percentages) (pooled results in Exhibit VI.16) 


T-Yo} xe F-] Michigan South Carolina 
=o [Ufer-laveyar-1melei cere) iil=) YC only YC only JC YC only JC 
Enrollment in school 
Within six months of YC 9 49 6 84 9 77 
One year following YC 8 10 6 11 11 8 
Two years following YC# 10 5 16 11 7 7 
Enrolled in two-year college 
Within six months of YC 7 49 5 84 9 77 
One year following YC 6 9 5 11 8 7 
Two years following YC# 8 2 15 11 4 4 
Obtained certification” 

Within six months of YC 0 17 - - 1 31 
Any time following YC 2 40 - - 4 53 
Sample size 383 105 174 108 123 91 


Source: National Student Clearinghouse sample weighted data. 


Notes: Analysis sample includes participants in YC Cohorts 4-6. Outcomes with very low probabilities 
were not included in this table due to small sample sizes. 


aQutcomes calculated two years following YC do not include YC Cohort 6. 


>Certification data were not available for Michigan participants. Sample is limited to Georgia and South 
Carolina. 
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Exhibit E.9. Criminal justice outcomes for JC and YC only participants (reported in 
percentages) (pooled results in Exhibit 6.17) 


Georgia WM Efedal(e FeTa) South Carolina 
Criminal justice outcome YC only YC only Jc YC only Jc 
Arrested for a new offense 
One year following YC 18 8 7 8 10 8 
Any time during follow-up 35 19 14 15 17 12 
Convicted for a new offense 
One year following YC 4 2 5 8 6 5 
Any time during follow-up 8 4 10 12 6 5 
Sample size 383 105 174 108 123 91 


Source: Criminal justice weighted administrative data. 


Note: Analysis sample includes participants in YC Cohorts 4—6. Rates represent the share of 
participants with a criminal justice event any time following the YC program. YC program end 
dates were considered to be May 31, 2017, for Cohort 4; December 31, 2018, for Cohort 5; and 
May 31, 2018, for Cohort 6. Criminal justice data were collected through June 2019 for South 
Carolina, November 2019 for Michigan, and December 2019 for Georgia. 
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To supplement the administrative data on criminal justice outcomes, we also collected self- 
reports of criminal justice outcomes through the follow-up survey. The follow-up survey 
contains data from participants collected for Cohorts 4, 5, and 6 beginning 18 to 20 months after 
the start of JC. This appendix presents the results of the follow-up survey questions related to 
criminal justice outcomes. Although these results provide interesting details on the 
characteristics of youth interactions with the criminal justice system, it is important to interpret 
these results with caution. First, youth might be reluctant to accurately report results on their own 
criminal justice involvement, either because of embarrassment or fear of repercussions. Second, 
youth might not be fully aware of the legal details of their cases or might accidentally 
misrepresent certain aspects of their situation. Finally, if youth are incarcerated or otherwise 
consumed by the justice system, they might have been unable to respond to the survey. Despite 
these limitations, results from the follow-up survey are close to estimates from the criminal 
justice administrative data on arrests (15 percent versus 16 percent, respectively) and convictions 
(6 percent versus 7 percent, respectively). 


Exhibit F.1. Self-reported criminal justice outcomes for JC participants (reported in 
percentages) 


Criminal justice outcome Percentage 


Arrested for a new offense 15 
Convicted for a new offense 6 
Convicted of a felony offense 4 
Sentenced to incarceration 6 
Sample size 150 


Source: Follow-up survey weighted data. 
Notes: Analysis sample includes JC participants in Cohorts 4-6. 
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Exhibit F.2. Self-reported charges associated with arrests and convictions for JC 
participants (reported in percentages) 


Percentage Percentage 

Charge arrested convicted 
Drug possession 3 2 
Selling or manufacturing of drugs 0 0 
Driving under the influence or driving while 0 0 
intoxicated 

Failure to pay child support 0 0 
Property offense 6 4 
Violent offense 2 1 
Other 2 1 
Sample size 150 150 


Source: Follow-up survey weighted data. 


Notes: Analysis sample includes JC participants in Cohorts 4—6. Not all respondents who reported 
having been arrested reported the associated charge. 


Exhibit F.3. Self-reported sentences for JC participants (reported in percentages) 


Sentence Percentage 


Fines 6 


Loss of driver’s license 1 
Mandated community services 2 
Probation 6 
Parole 0 
Sample size 150 


Source: Follow-up survey weighted data. 
Notes: Analysis sample includes JC participants in Cohorts 4-6. 
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In this appendix, we examine whether key results are sensitive to the definition of court 
involvement. The analysis in the main body of the report uses a measure of court involvement 
constructed from information that youth self-reported on the background information form.*” 
Using the background information form enables us to have a well-defined and consistent 
definition across sites. Using this approach, we find that 41 percent of participants have court 
involvement, as shown in Exhibit G.1. An alternative approach would be to use the indicators of 
court involvement that sites provided in the administrative program data. Michigan and South 
Carolina provided court-involvement indicators for all YC participants, while Georgia only 
indicated court involvement for JC participants. Using site definitions, 30 percent of participants 
are found to have court involvement. 


Exhibit G.2 shows the share of youth who are categorized as court-involved under each 
definition. Overall, 75 percent of youth were categorized consistently across the two definitions. 
Nine percent of youth were categorized as court-involved according to the site definitions but not 
according to the study definition. This might reflect differences in what was considered court- 
involved or youth’s underreporting of involvement in the criminal justice system. Likewise, 16 
percent of participants were categorized as court-involved according to the study definitions but 
not according to site definitions. 


Exhibit G.1. Court-involvement rates, by definition and site (reported in percentages) 


Source of court-involvement T-Yo} ce fF] Michigan South Carolina 
Study defined court-involved? 41 44 37 39 
Sample size 963 477 278 208 
Site defined court-involved> 30 56 27 23 
Sample size 555 78 292 185 


Source: Background information form weighted data and program administrative data. 


Notes: Analysis sample includes YC participants in Cohorts 4-6 who completed the background 
information form. Youth were categorized as having court involvement by the study definition if at 
baseline they reported ever being arrested, found guilty of a status offense, convicted of a crime, 
spent time in a juvenile or adult detention facility, or if they were on probation or parole at the time 
of entering YC. 


aAnalysis limited to participants with a nonmissing indicator of court involvement using the study 
definition. 


bAnalysis limited to participants with a nonmissing indicator of court involvement using the site definition. 


37 Youth were categorized as having court involvement by the study definition if at baseline they reported ever 
being arrested, found guilty of a status offense, convicted of a crime, spent time in a juvenile or adult detention 
facility, or if they were on probation or parole at the time of entering YC. 


139 


Appendix G Mathematica 


Exhibit G.2. Comparison of court-involvement rates, by definition (reported in 
percentages) 


YoU] qex: Meo) merelUlumiranceyhV{clanl-Jais Site court- involved ATi <cM ale) mexel UT atom lance) hV{-1e| 
Study defined court-involved 21 16 
Study defined not court-involved 9 54 


Source: Background information form weighted data and program administrative data. 


Notes: Analysis sample includes YC participants in Cohorts 4-6 who completed the background 
information form and had nonmissing indicators of court involvement by both definitions. Youth 
were categorized as having court involvement by the study definition if at baseline they reported 
ever being arrested, found guilty of a status offense, convicted of a crime, spent time in a juvenile 
or adult detention facility, or if they were on probation or parole at the time of entering YC. 


To illustrate the impact of these differences, we estimate the key outcomes by court involvement 
using the site definition (Exhibit G.3). Many differences in outcomes between court-involved 
participants and not court-involved participants are larger according to the study definitions than 
the site definition. For example, there is a 14 percent difference in post-program arrests between 
court-involved and not court-involved participants but only a 2 percentage point difference by 
the site difference. 
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Exhibit G.3. Select JC participant outcomes, by court involvement (reported in 
percentages unless otherwise specified) 


Study definition? Site definition 

Court- Co) exe 01 ate Court- Co) exe 01 ate 
Outcomes Tanyce) NYZ=xe | Tanyce) NYC=re | ance) WYZ=xe| ance) hYC=xe | 
Education 
Received any college credit® 70 78 74 179 
Currently enrolled in college® 4 10 6 8 
Currently enrolled in any courses4 22 30 21 29 
Employment 
Currently working 78 82 85 81 
Average weekly earnings? 394 374 391 361 
Average weekly earnings 508 456 462 446 
conditional on working‘ 
Military 
Currently enlisted4 10 16 4 17 
Any productive activity4 81 88 85 86 
Crime and delinquent activity 
Arrested® 25 11 16 14 
Convicted® 13 4 14 8 
Sample size (NSC and CJ) 103 194 46 144 
Sample size (FUS) 53 95 39 74 


Source: Background information form weighted data and program administrative data. 

Notes: Analysis sample includes YC participants in Cohorts 4-6 who completed the background 
information form. Youth were categorized as having court involvement by the study definition if at 
baseline they reported ever being arrested, found guilty of a status offense, convicted of a crime, 
spent time in a juvenile or adult detention facility, or if they were on probation or parole at the time 
of entering YC. 

aAnalysis limited to participants with a nonmissing indicator of court involvement using the study 

definition. 

>Analysis limited to participants with a nonmissing indicator of court involvement using the site definition. 

¢Source: National Student Clearinghouse (NSC) weighted data. Reflects outcomes two years following 

YC completion. 

4Source: Follow-up survey (FUS) weighted data. 

®Source: Criminal justice (CJ) administrative weighted data. Reflects arrests and convictions any time 

following YC completion. 
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